When to #OpenSource or Not

If you’re a developer and have no intentions of ever getting paid for writing code, then deciding between open-sourcing your software or not can be a simple decision. However, if you have any intentions to get compensated for your work, then the decision to open-source or not can be a complicated and long-winded debate which has gone on for decades. On the one side, you have the capitalists/conservatives that sell software for a price and their pricing models are relatively simple to understand especially in commercial and non-enterprize environments i.e. you get this product for this price. On the other hand, you have the open-source socialists/liberals/anarchists, advocating for making software “open”, transparent and most of the time offering it for free. However, for anybody that has ever delved into the open-source debate, they will quickly realize that open-source is in fact not that clear and transparent after all.

It’s all good and well to spew out socialist rhetoric about how we should all be more altruistic by volunteering our time and skills to open and transparent software communities, but once we’ve all had our turn at singing kumba-ya, the truth is that in reality there’s no such thing as free lunch. Meaning that although many open-source enthusiasts pitch the idea of so-called free software, the users will still end up paying for it one way or another, be it in some sort of fee, time, energy and/or frustration. The reason being that in the real world, time equals money and money make the world go round. Therefore, any developer working on a “free” and open-source project still needs to pay the bills, put food on the table and at the very least still have money left over to buy a laptop on which to work on. So how do open-source developers get compensated? Open-source projects typically start off as side-line hobby projects, then perhaps moving on to offering paid for versions and/or charging for services.

Over the years I’ve thought about this debate the countless number of times and like most analytical people I’ve searched far and wide for an absolute answer i.e. is open-source good or bad. When I was younger I leaned more towards the open-source point of view and their anarchist/hippie rhetoric about how corporations are evil, money-grubbing establishments designed to create monopolies and control people’s lives. Now that I’ve gotten older, my view has become a bit more conservative. Maybe it’s got something to do with there being a grain of truth in Winston Churchill’s wise words where he said that “any man who is under 30, and is not a liberal, has no heart; and any man who is over 30, and is not a conservative, has no brains.”

So here’s my attempt at an absolute answer as to when to open-source and when not to:

  • How most open-source projects start off: as the saying goes, “necessity is the mother of all inventions”. By that token, most open-source projects start out from a need/problem that someone has. A developer may want to achieve something that they cannot do with commercial software that is available or they just don’t have the money to purchase commercial software that can perform the given task. While developing this new app for their own needs, it suddenly dawns on the developer that other people may have the same need/problem that their new app can solve. At this point, the developer begins to think about how they could possibly get others to use this new app and possibly capitalize on it. After careful consideration they realise that there’s more to selling software than simply putting a price tag on it i.e. a company needs to be set up, marketing needs to be done, payments need to be captured, orders and invoices need to be created, taxes need to be paid and amongst many other things, all sorts of people need to be hired e.g. marketing, sales, finance, support etc. Starting and managing a company requires a lot of work and responsibility. Due to the fact that a decent amount of capital is required, it also involves a huge amount of risk i.e. it’s not always easy to predict whether or not a software product will be a hit or not. So as opposed to going through that whole nightmare, our friendly developer decides to rather just upload his code to a popular open-source repository that all his friends can access. If nothing else, it will look good on their resume.
  • If the software is a hit: if there proves to be a substantial demand for the software, the developer may begin thinking about capitalizing on it. At this point, the developer will probably be stuck between a rock and a hard place. If he closes down the open-source repository and begins to charge for the software he’s fear is that the users will just continue using the latest open-source version available and never purchase any new versions. Even if there are some people willing to pay for the software, the problem is that the developer will never be able to sell it to enterprise customers because no large company will agree to adopt a product that is being supported by a single individual i.e. it would just be too risky for the company. So it’s a bit of a chicken and egg scenario i.e. you need to generate revenue to hire more people, but customers are hesitant to commit to a one-man show. On the other hand, if he keeps the open-source repository available, then there is no way to make money on it. To get out of this dilemma, most developers in this situation either begin selling services or offering paid for upgraded versions with additional features.
  • Charging for services approach: most open-source developers will write the code for free to develop the software, but if users need any help with specific feature requests, installations, advice or any consulting services, they will be charged an hourly rate or are sold a 1/3 year Service Level Agreement. As a bit of a realist, it is at this point that I start questioning the incentives put in place for this type of revenue stream. I ask myself, what incentives do open-source developers have to produce quality software that is an easy install and use when their primary revenue stream comes from selling services? If charging for services is your only revenue stream on an open-source project, then there is a clear conflict of interest in this approach and I would advise against it. Reason being that what users want is software that is easy to install, configure and use, while at the same time the developers are well aware that the easier the software is to configure and use, the fewer services they will sell. So what you end up with, is open-source software that is counterintuitive and users are constantly left with more questions than answers. In this kind game with these kinds of incentives in place, there is not much that is open and transparent to the users i.e. the software and code are available and free for anybody to use, but nobody knows how to set it up and use it. You may be of the opinion that it doesn’t matter whether there’s a conflict of interest or not, as long as you’re making some money off of your efforts. However, my argument would be that users are not stupid, they always know when there is a conflict of interest and sooner or later they will start posing questions. This approach also creates confusion that results in users asking why you’re doing some work for free while charging for other work. This is where open-source can lack transparency. With the above said, there are some more negative side effects with this approach:
    • Minimal documentation: the whole point of writing documentation is to make it easier for users to understand the process of installing, configuring and using a given piece of software. Given the fact that many open-source software is offered for free, the developers have almost no incentives to write comprehensive documentation i.e. the more difficult it is for a user to get up and running and troubleshoot issues themselves, the more services the developers can sell. What you naturally end up with is open-source software that often has little to no documentation aside from a simple two-page quick start guide. The counter-argument from open-source developers is that “the code is the documentation, just look at the code”. But if we cross over from the delusional hippie world into the real world, we realize that most users are not coders and therefore having the code adds zero value to their lives. Even as a coder myself, if I’m interested in using someone else’s software I will neither have the time nor the interest in doing support on other people’s products. Personally, I just want to be able to click install, use the software and move on with my life because I have enough of my own code to deal with, without having to deal with other people’s broken code.
    • Lower quality of open-source software: in terms of quality, the argument made by open-source purists is that their software is of much higher quality than that of proprietary software. According to them, this is due to the fact that open-source communities are larger and there are more people testing, spotting bugs and fixing them as opposed to small teams that develop proprietary software. This is based on Linus Torvalds’ law that “given enough eyeballs, all bugs are shallow”. That does indeed sound plausible, but on the other hand, it seems to me like a bit of a contradiction to the many open-source companies that rely on services as their primary revenue stream. If the quality of open-source software is so incredibly high, then how do they make their money on services i.e. if there are no bugs to fix, the apps are easy to use and everything just works, then what kind of services are these developers selling to pay the bills? Developers creating proprietary software, on the other hand, have every incentive to write high-quality code and create intuitive user interfaces, because if they don’t, then nobody will buy their software.
  • Paid for upgraded versions approach: an alternative to only offering services is to develop additional features into the open-source software and offer those features at a premium i.e. offering a “free” open-source community version as well as a paid for version with additional features that are enticing enough that make users want to spend money. Open-source can be fantastic when you want to lower your risk while at the same time increase the user adoption rate of your software. By open-sourcing your software you can start a project without having to take on the risk of starting a company, investing capital etc. and if there proves to be a high enough demand for your software, you can then take it to the next level by offering additional features and charging for them. This is the only time I would see open-source as good idea i.e. assuming that you actually want be compensated for your work.

In summary, before open-sourcing your software you should consider the number of features that you have in mind and whether or not there will ultimately be a demand for those features on the market. There are many apps that can be incredibly useful, but beyond their basic features, there are a limited number of additional features that can be developed and incorporated into future paid for versions. A great example would an SDK (Software Development Kit) to interface with third-party application or hardware: there are only so many features you can add to such an SDK. This kind of software with limited useful features should either be open-sourced without any intention of ever making money or it should never be open-sourced, but rather be released and sold as proprietary software right from the start.

Advertisements

Uniform Interfaces in a Nutshell

The uniform interface constraint defines the interface between clients and servers. It simplifies and decouples the architecture, which enables each part to evolve independently. The four guiding principles of the uniform interface are:

Resource-Based

Individual resources are identified in requests using URIs as resource identifiers. The resources themselves are conceptually separate from the representations that are returned to the client. For example, the server does not send its database, but rather, some HTML, XML or JSON that represents some database records expressed, for instance, in Finnish and encoded in UTF-8, depending on the details of the request and the server implementation.

Manipulation of Resources Through Representations

When a client holds a representation of a resource, including any metadata attached, it has enough information to modify or delete the resource on the server, provided it has permission to do so.

Self-descriptive Messages

Each message includes enough information to describe how to process the message. For example, which parser to invoke may be specified by an Internet media type (previously known as a MIME type). Responses also explicitly indicate their cache-ability.

Hypermedia as the Engine of Application State (HATEOAS)

Clients deliver state via body contents, query-string parameters, request headers and the requested URI (the resource name). Services deliver the state to clients via body content, response codes, and response headers. This is technically referred-to as hypermedia (or hyperlinks within hypertext).

Aside from the description above, HATEOS also means that, where necessary, links are contained in the returned body (or headers) to supply the URI for retrieval of the object itself or related objects. We’ll talk about this in more detail later.

The uniform interface that any REST services must provide is fundamental to its design.

Advantages of Cloud Computing

We live in a modern, fast-paced world where people want to have all things to be instant. Through the internet, it makes our work faster and convenient. For example, you can have remote access to your files and applications using the internet and it frees you from using storage devices like flash drives. Whereas in the past, people would run applications or programs from software downloaded on a physical computer or server in their building. With cloud computing, it grants people to access to the same kinds of applications through the use of internet.

 

According to a set of reporters tackling cloud computing, cloud computing is a concept that works hand in hand with mobile computing in order to truly achieve computing on the go. This basically grants users or consumers to have remote access to data, applications, and storage for convenience and frees them from physical hardware needs.

 

In other words, it is a delivery system that delivers computing. Examples of cloud computing include most of Google’s services like Gmail, Google Calendar, Evernote, Google Maps and so on. For me, cloud computing is a big help, especially that I am a busy person and I can access my files anytime and anywhere. This is a great use either for business or personal means. Cloud Computing is mainly categorized into three services including, software as a service (SaaS), infrastructure as a service (IaaS) and platform as a service (PaaS). Key characteristics of Cloud Computing are on-demand network access, scalability, greater security, measured service, and flexibility. The reporters also shared their reasons why there are several people who move to cloud computing. They mentioned a lot of benefits including unlimited storage capacity, device independence, increased efficiency, increased data reliability, lower computer cost, easier collaboration, etc,. Despite a number of benefits, there are also its disadvantages including it requires a constant internet connection, does not work well with a low-speed internet connection and has possibilities of data being lost or not secured.

Cloud Computing is one of the most popular breakthroughs in IT today. The “cloud” represents one of the most significant shifts that computing has gone through. I think that although cloud computing is lacking a bit in some aspects and needs improvement it is a new model and architecture so it still has an opportunity for future research and expansion.

Nowadays, it is clear that cloud computing has revolutionized how technology is obtained, used and managed. And all of this has changed how organizations budget and pay for technology services.

Cloud Computing has given us the ability to reconfigure quickly our environments to be able to adapt them to changing business requirements. We can run cost-effective services that can scale up and down depending on usage or business demands and, all of this, using pay-per-use billing. Making unnecessary huge upfront infrastructure expenses for enterprises, and balancing the possibilities of being a big enterprise or a new one.

 

google-may-be-winning-some-big-cloud-customers-but-it-has-a-long-way-to-go-to-unseat-amazon

There are multiple and diverse advantages, and most of them depend on the enterprises, the business and the needs they have. But, there are six of them that tend to appear in every case:

Variable vs. Capital Expense

Instead of having to invest in data centers and servers before knowing how they are going to be used, they can be paid when you consume computing resources and paid only for how much you consume.

Economies of Scale

By, using cloud computing, we can achieve a lower variable cost that we would get on our own. Cloud providers like AWS can aggregate hundreds of thousands of customers in the cloud, achieving higher economies of scale, which translates in lower prices.

Stop Guessing Capacity

When we make a capacity decision prior to deploying an application, we usually end up either setting expensive idle resources or dealing with limited capacity. With cloud computing, there is no more need for guessing. The cloud allows us to access as much or as little as we need to cover our business needs and to scale up or down as required without advanced planning, within minutes.

Increase Speed and Agility

Deploy new resources to cover new business cases or implementation of prototypes and POCs to experiment now can be achieved with a few clicks provisioning new resources in a simple, easy and fast way usually reducing costs and time and allowing companies to adapt or explore.

Focus on Business Differentiators

Cloud computing allows enterprises to focus on their business priorities, instead of on the heavy lifting of racking, stacking and powering servers. This allows enterprises to focus on projects that differentiate their businesses.

Go Global in Minutes

With cloud computing enterprises can easily deploy their applications to multiple locations around the world. This allows providing redundancy and lower latencies. This is not any more reserved just for the largest enterprises. Cloud computing has democratized this ability

#Recommendations as an #API: what is a Recommended System Nowadays?

 

What is an API?

An API is an application program interface which has determined routines, protocols, and tools for creating application. APIs are used to program interface components and they’re important because they provide the building blocks for making a program, which are then put together and completed by a given programmer. An API essentially makes the programmer’s job easier and less stressful.

Types of APIs

            There are multiple kinds of APIs for different operating systems and for different purposes that suit the needs of different people. Let’s take Windows, which has several APIs that are used to run hardware systems and applications. The operating environment of most systems provide APIs and this allows programmers to create applications or programs that are compatible and consistent with the operating environment. Nowadays, however, there are websites that utilize APIs as well; for example, both Amazon and Ebay use APIs which allow programmers and developers to use existing retail infrastructure to create web stores. Also, third-party developers are able to use APIs on the web to solve problems with software for end-users.

0006.jpg

Examples

There are many examples of APIs and the ProgrammableWeb tracks over 15 500 APIs. It also lists Google Maps, Twitter, Youtube, and Flickr as a few of the very most popular APIs.

SaaS Recommenders

            SaaS is an acronym for software as a service recommender. These systems have several challenges and these include the handling of multi-tenancy. Multi-tenancy is a term for software architecture that allows a single instance of software to run on a server for multiple tenants, who are users that share access to software instance. SaaS Recommenders also process a large amount of data and they keep client’s data safe. Essentially, recommender systems work to filter data in order to determine the rating or preference a customer would give to an item. These systems are very common nowadays and they’re used in many areas of trade. There are recommender systems which are open-sourced, academic, media-centered, etc. It all depends on the niche you’re looking for!

Recommender systems work through collaborative filtering and content-based filtering. Collaborative filtering suggests items based on a client or visitor’s past used items and takes into consideration other users who chose similar items. This kind of filtering determines items that a user might be interested in. Content-based filtering, on the other hand, gives suggestions based on items with similar characteristics of a specific item so that the system can suggest items with similar properties.

The benefit of using SaaS Recommender Systems resides in your being able to pay for value with a low overhead instead of having a large upfront investment. SaaS Recommenders generally have a clear path to use and they provide with development and improvement as you use them. Below, you can find a list of Recommender Systems:

  1. Suggest Grid, which is the descendant of Rcmmdr, personalizes API for developers and recommends items based on the user’s previous items. The system uses collaborative filtering and allows visitors to see recommendations that they otherwise wouldn’t. Overall, it’s a generic system that is still highly suggested.
  2. Mortar Recommendation Engine is one that allows the programmer to create his or her own system through their PaaS Mortar and MongoDB’s. This system is used by MTV, Associated Press, etc. It’s made to be flexible, open, and portable.
  3. Peerius is focused on e-commerce focus for live and email recommendation. Like other systems, it works to personalize API and works with more the two hundred and fifty retailers. Individuals are exposed to product suggestions, messages, etc. and Peerius has been proven to increase sales.

There is also a nice list of such engines available to look at.

Conclusion

APIs are essential in web development; they’re an interface that specifies routines and commands that allow programmers to create programs. Above, you were introduced to APIs, different kinds of APIs and SaaS Recommenders.

Cultural Heritage #APIs: What are they, why we need them

Introduction

Currently, it is of very great significance for the cultural heritage scholars to highly focus on the area of application programming interface (APIs). Generally, APIs are considered to be in very simple terms, code libraries that are assembled by various companies offering web service with a major objective of enabling third-party applications to be able to link up and make communications with web service platforms[1]. To define cultural heritage APIs, it refers to an expression of different ways of living that are normally developed by a community, and then passed from one generation to the other comprising of practices, customs, values places, and artistic expression achieved through the help of technology.

Usually, API is invisible to the human eyes regardless of the fact that APIs are interfaces that are linked to facilitate computer-to-computer communication. Inside the domain of cultural heritage, there is a very implausible potential create tools that can help in revolutionizing both the presentations of different collections and the way in which people happen to experience and interact with cultural heritage. Cultural heritage APIs are of very great significance giving the reason as to why we need them. Taking, for example, it is through cultural heritage APIs that we are able to discover the Europe’s rich cultural heritage that can be found in the museums, libraries, galleries, cultural oriented institutions, and different archives in the whole content of Europe. In addition, it is through the digitization efforts that enables people from different parts of the world to get to know about this heritage through different online platforms[2]. This simply means that cultural heritage APIs helps in the preservation of the important cultural materials.

Museum REST APIs

In application program interfaces, REST is used as the type of architecture style that is meant for the design of networked application whereby there is a use of simple HTTP in the making of calls between machines. There are different categories of museums APIs whereby they are all designed by implementing REST so that they can provide the expected purpose when it comes to the cultural heritage. These categories comprise of arts, education, and location. In the category of arts, museums APIs aim at integrating the entire museum’s collection into an application hence allowing different users to have access to data concerning objects, people, exhibitions, galleries, and publications. Under the education category, museums APIs are designed as a REST-ful interface depending to a particular museum’s collections so that all the items searched can be returned in the database while they are already paginated in either JSON or SML format.

Availability of Museums REST APIs

Looking at the availability of museums REST APIs, in most cases, it involves the use of Gerrit code review which normally comes with a REST-like API that is always available over the HTTP. In this case, the API happens to be very suitable when it comes to the automated tools to build upon and also supporting a number of ad-hoc scripting uses cases. It is through the construction of the API protocols as well as the documentation that different web service provider companies enable people to have access to the data in regard to cultural heritage they may want to access.

The Necessity of Museum APIs

There are a number of significances linked to the museum APIs in regard to the cultural heritage. To start with, use of Artsy API, it helps in providing access to images of a particular historic artwork as well as related information when it comes to artsy being one of the category of museum APIs meant for educational and some other non-commercial purposes[3]. The other significance is that it helps in providing access to positions, luminosity, color, and some other data to people in different locations of the world when it comes to the exoplanets and constellations. Allowing an individual to search a diverse body of online primary resources that relates to written and early printed culture in a particular state, taking for example of the Britain culture at the period of 1000-1500, is the other necessity of museum APIs lying under manuscripts online API. The other significance is on the connected histories API whereby it brings together a range of digital resources when it comes to the early modern as well as the 19th century Britain with a single federal search that normally enables a highly modernized searching of names, places, and dates, connect, and shares resources with a private workspace. Usually, the connected histories APIs allow different users to connect programmatically when it comes to the search engine through the use of GET parameters whereby results can be retrieved in an XML format[4].

Future Thoughts of Museum REST APIs

The future thoughts of the museum APIs involves determining where next to open cultural data in museums[5]. It is evident that recently, museums have increasingly been integrating the global movement when it comes to the open data through initiating of their databases, images sharing as well as releasing of their knowledge. The other future thoughts involve determining on how to come up with a way that can be used to in open data so that there can be engaging of more people and more diverse individuals taking, for example, of the United Kingdom heritage and culture.

Cloud Technologies Used by Museums

Cloud technology or computing can be defined as a natural progression when it comes to the utility of computing. Normally, early computers would require a number of users to share a single console which has currently come to an end as the advent of personal computing brought about the convenience into our homes[6]. The Internet has completely changed the way in which people link up to information and each other. Museums are known to provide access to their different collections and programs through the hosting of websites as well as the applications for the public use.

Museums make use of software as a service (SaaS) and Platform as a service (PaaS) as its main types of cloud computing[7]. For the PaaS system, they normally enable access when it comes to the virtual hardware devices that always allow the software toolsets that can be most appropriately used. SaaS systems normally require users to deploy software as per a particular interface with a major objective of hosting applications[8].

Through providing of API only access, the implementation of SaaS can have the capability of providing features that can help developers to build upon considering the aspect of scalability as the most significant one. Consenting the cloud to take control of where the data gets to be stored and where the applications happen to be stored through a SaaS system helps in removing a significant amount of workload as well as the complexity of the developer. For the PaaS, museums consider using it due to its benefit of ease in moving applications from one servicer provider to the other. Museums considered there switching to the cloud computing technology due to these operational advantages linked to the cloud computing technology and environmental effects through the use of services in the cloud[9].

When it comes to the data exposed by the cloud technologies used by the museum’s work, they normally get to be used by some other museums in the worldwide basis so that they can be able to advance their museums work with a major objective of preserving the cultural heritage in different parts of the world.

REST APIs are considered to be very significant and useful due to the fact that they do allow different users to have access to data portals when it comes to different museums natural history with an objective of retrieving the collection and research datasets for different uses in software or applications. Talking, for instance, of the London natural history museums, the datasets get returned in JSON whereby it holds over 2 million specimen records that are from the museum’s zoology, botany, and entomology collections.

 

 

Bibliography

Armbrust, Michael, Armando Fox, Rean Griffith, Anthony D. Joseph, Randy H. Katz, Andrew Konwinski, Gunho Lee, David A. Patterson, Ariel Rabkin, Ion Stoica, and Matei Zaharia. Above the Clouds: A Berkeley View of Cloud Computing. Berkeley, CA: Electrical Engineering and Computer Sciences University of California at Berkeley, 2009. Accessed September 6, 2016. https://www2.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-28.pdf

Fernando, Niroshinie, Seng W. Loke, and Wenny Rahayu. “Mobile cloud computing: A survey.” Future Generation Computer Systems 29, no. 1 (2013): 84-106.

Isaksen, Leif. Pandora’s Box: the Future of Cultural Heritage on the World Wide Web. (2009): 110-130. Accessed September 6, 2016. https://leifuss.files.wordpress.com/2008/08/pandorasbox.pdf

Johnson, Larry, Samantha Adams Becker, Victoria Estrada, and Alex Freeman. The NMC Horizon Report: 2015 Museum Edition. Austin, TX: The New Media Consoritum, (2015):190-205.

Rosenthal, Sara, Alan Ritter, Preslav Nakov, and Veselin Stoyanov. “Semeval-2014 Task 9: Sentiment Analysis in Twitter.” In Proceedings of the 8th International Workshop on Semantic Evaluation, 73-80. Dublin, Ireland: SemEval, 2014.

Sucan, Ioan A., Mark Moll, and Lydia E. Kavraki. “The Open Motion Planning Library.” IEEE Robotics & Automation Magazine 19, no. 4 (2012): 72-82.

 

 

[1]Michael Armbrust, Armando Fox, Rean Griffith, Anthony D. Joseph, Randy H. Katz, Andrew Konwinski, Gunho Lee, David A. Patterson, Ariel Rabkin, Ion Stoica, and Matei Zaharia. Above the Clouds: A Berkeley View of Cloud Computing. (Berkeley, CA: Electrical Engineering and Computer Sciences University of California at Berkeley, 2009), 530.

[2] Leif Isaksen. Pandora’s Box: the Future of Cultural Heritage on the World Wide Web. (2009), 120.

[3] Sara Rosenthal, Alan Ritter, Preslav Nakov, and Veselin Stoyanov. “Semeval-2014 Task 9: Sentiment Analysis in Twitter.” In Proceedings of the 8th International Workshop on Semantic Evaluation (Dublin, Ireland: SemEval, 2014), 73-80.

[4] Ioan A. Sucan, Mark Moll, and Lydia E. Kavraki. “The Open Motion Planning Library.” IEEE Robotics & Automation Magazine 19, no. 4 (2012): 72-82.

[5] Larry Johnson, Samantha Adams Becker, Victoria Estrada, and Alex Freeman. The NMC Horizon Report: 2015 Museum Edition. (Austin, TX: The New Media Consoritum, 2015),192.

[6] Ibid., 200.

[7] Niroshinie Fernando, Seng W. Loke, and Wenny Rahayu. “Mobile cloud computing: A survey.” Future Generation Computer Systems 29, no. 1 (2013), 90.

[8] Ibid., 100.

[9] Ibid., 102.

Exploring Cache options for Hypermedia #APIs

Application program interface (API) is a set of protocols utilized for building software applications. This directs how the components of software interact with one another and are very useful as they make it easier to develop the program and put all the units together. More recently Hypermedia APIs have become the latest hype, scaling better and promoting decoupling and encapsulation easily along with the myriad of advantages these things bring.

REST (Representational state transfer) is a popular architectural approach to communications that currently utilized in the development of web software. REST’s lighter communication between producer and consumer makes it a well-liked system for cloud-based APIs, especially those provided by Google, Mircosoft etc. Running over the HTTP (hypertext transfer protocol, this architecture has several components, one of the most important being able to control a cache. REST APIs operate in a simple manner, by picking up a base protocol like HTTP and model the resources with a URI. You can then map actions into HTTP methods. However, hypermedia augments this procedure by simultaneous presentation of information and control hence poses as a pivotal constraint of the web and as an extension of REST.

So what is Hypermedia control?

As the digital world exceeds in complexity, more and more people want different components of a system all within a single application, to be able to easily interact with one another clearly. This cross-linguistic data exchange is what APIs have always been evolved to do, however, languages used to code individual software applications do not correspond nicely with each other.

Hypermedia that is hyperlinked can circumvent this need and can cause applications to stay interconnected and communicate easily. Hypermedia APIs is a more evolved and phenomenal way of creating APIs by using the universally used and understood hyperlinks.

This hypermedia functionality can embed links into associated resources in order to describe the action it will perform. This allows end users to carry out the actions that have been prescribed in the resource.

Current situation of web caches:

Due to the dynamic nature of web application, a lot of data is required to be stored. With every new request HTTP, caching is needed. With this caching, you cache the full output of the page requested, bypassing your application entirely on succeeding requests. While this is the current situation of web caches, it isn’t always possible for highly dynamic sites, which can only use the power of HTTP caching on fragments on these sites.

Caching with Hypermedia APIs:

Of the major issues with APIs and web-based architecture on a network is that optimum performance is acquired when the network is not in use, as this is the dawdling part of any application. Hypermedia APIs however, allow a greater caching of data on the browser cache through two ways, either a content delivery network or through a proxy on the server. This altogether reduces dependency on the network server infrastructure, hence increased speed considerably.

In addition, there is also a possibility of remote procedure call (RPC) with hypermedia APIs, where dynamic elements can be edge cached. In certain popular websites, this allows the trendy products or articles to be updated frequently along with the comments, opening a more interactive website.

One of the keys to successful Hypermedia API is its use of caching such as local caches and Etags. Moreover, in large systems cache invalidation can prove to even more costly than not caching at all. However, through careful designing of hypermedia resources, you can easily enable your system to depend on natural behaviors that are built into HTTP protocols.

Other benefits of Hypermedia APIs:

There are other benefits of Hypermedia APIs as well. These can enable applications to browse an API from the root or any resource, acting in a way that web works with hyperlinks. This needs API federation, where media APIs are interlinked with content API.

Moreover, it allows for more flexible and evolvable systems, as URIs remain persistent or unchanged, all configurations are done at the API root. If under any circumstance, the API is faulty, no damage will be suffered by the URL infrastructure and hence, this proves to be very cost effective.

In addition, Hypermedia APIs are also extensible. This means that more links and forms can be added without the fear of API breakage. These will also simply be ignored if they are not supported by the application of the client. Hence, validation and guidance of client are ensured.

Hypermedia APIs are simple and enable scalability from small to large single-page apps. They also enable standardization when API calls and made and the information if returned. The various links and forms are affordances in the hypermedia infrastructure and can be directed after the validation from the client side. While this advantage is still in research phase, it can prove to be very beneficial for certain businesses.

Another subtle advantage of hypermedia APIs over traditional APIs without hyperlinks is that in the latter you can only expose information that the client has requested, not caring what it is being utilized for. In a Hypermedia API, you are conscious of the workflow and can actively guide the client by providing links. This creates a superb communication between clients and server developers.

Limitations of hypermedia APIs:

Despite the obvious benefits, Hypermedia APIs still have certain drawbacks; however, these are just certain barriers that have to be climbed to allow for the adoption of this wonderful technology.

One of the most critical arguments against Hypermedia APIs is that the payload is higher for serialization of JSON, unlike thrift objects. However, overall if API actions are taken copiously, less data needs to be transported, so procedures like call sessions might not increase by such a high amount.

Future of Hypermedia APIs:

As systems become even more intricate, the need for universal language among diverse systems will be needed. Hypermedia, despite its limitations, will be the panacea in overcoming such communication gaps, especially in areas where Big Data and caching is concerned. Hypermedia is powerful and its full potential has yet not been discovered. It might even someday leave today’s REST and HTTP methods behind, relying solely on its ability to link through the common protocol, unorganized distributed data. How it completes this task only time will tell.

Scaling Celery #Heroku Workers with Hirefire.io

Recently I had to heavily use background workers because of the increasing traffic on Orfium. This caused me a little bit of panic because I am mostly an API/Rest guy and not of such a favour of MQTT. I have to admit I was wrong. But let’s take it from the beginning.

Proudly Deploying our API

For Orfium, our traffic occasionally spikes and we need to scale up quickly. Heroku is great for this, but it requires that one of us is always watching the site. For months we’ve been manually scaling up and back down. Occasionally we’d scale up and then forget to back off on the dynos (web servers) when traffic went back down. It’s wasteful to have those extra dynos idle and costs us a lot of money.

Right now, on our platform we have our first contracts with some distributors and that means that we needed to create a new pipeline for them to upload hundreds of thousands of tracks, albums and playlists!

As an experienced API guy, I designed our API and actually since we have had super specific flows we needed to implement, we used django-rest-framework for providing us the tools we needed.It took us less than a week to have a stable prototype to start evaluating with our first requests.

All tests passed with success and everyone was soooooo happy!

Proudly un-Deploying our API – Long Live MQTT

Then, real life started happening.. Distributors started deploying either super many tracks at once or just one track and the traffic was totally irrelevant. We couldn’t predict this and many http requests started happening on our servers which starting making our server exploding… It never crashed, but NewRelic Apdex score was not happy at all.

With my fellow hacker, Dimitri, we agreed we needed something permanent. We rewrote most of the code, and started processing everything with Celery workers. Obviously we still use tones of APIs, but this couldn’t have happened without MQTT and celery.

cb2fc8174648f5862d73efc33109e8ea_400x400

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.The execution units, called tasks, are executed concurrently on a single or more worker servers using multiprocessing, Eventlet, or gevent. Tasks can execute asynchronously (in the background) or synchronously (wait until ready). Celery is used in production systems to process millions of tasks a day.

Now NewRelic provides us with complete analytics for each background process and the overall server load has been dramatically reduced.

Reducing Costs – Improving Scalability and Availability

I know this may sound a little bit provoking but it couldn’t be more true. The specific miracle has a also a name.

It is called, Hirefire.

We have been using it the last month for our web dynos, and we already paid less than half on dynos and now it takes care of our workers as well!

Hirefire is also also OpenSource which means that you can host it anywhere you like, but myself and many others prefer the hosted solution directly.

In this blog post I will focus only on how to scale celery workers, but in any case, the guys are awesome and they have the BEST CUSTOMER SUPPORT EVER!! (thank you michael)

Some Python

Let’s dive-in a little bit on some code.

First, let’s assume you have a django app (or not) already set up with celery.

Then you just need to install HireFire to automatically scale your worker dynos.

This is as simple as:

# pip install hirefire

from celery import Celery

from hirefire.procs.celery import CeleryProc

celery = Celery('myproject', broker='amqp://guest@localhost//')

class WorkerProc(CeleryProc):
    # the name field should be identical with the worker name on your
# Procfile!! It is nowhere explicitly written,
# but otherwise it won't work!
    name = 'worker'
    queues = ['celery']
    app = celery

You can find more information on the corresponding docs. It is not difficult to setup, but it needs to be a little bit careful.

If you follow the instruction README of the lib, under the Django section (or any section that works for you),  you can test to see if your configuration works in development by accessing http://localhost:8000/hirefire/development/info . When you access this URL, you should see a JSON object in your browser containing the configured queues and their sizes (try enqueuing a few jobs and refresh this url to see if the sizes increase).

A brief description of how it works:

Once per minute, HireFire will perform an HTTP GET request to https://your-domain.com/hirefire/HIREFIRE_TOKEN/info to fetch your current queue size(s). This, combined with whatever you’ve configured in the HireFire web interface for your worker manager (Worker.HireFire.JobQueue), will determine how to scale. So basically the only thing the Python lib does is it exposes a route which HireFire will access to retrieve your configured queue sizes.

Basically what you want to do is the following:

  1. Log in to HireFire
  2. Add a new manager for your app with the name worker (assuming your worker is also named “worker” in your Procfile) of type Worker.HireFire.JobQueue
  3. On the app overview page in the HireFire UI you should see a row labeled “Token”. Click it and it’ll give you a command to run in the terminal to add the token to your app
  4. Add the Python library to your app and configure it
  5. Test to see if JSON is returned when accessing localhost:8000/hirefire/development/info — add some jobs and refresh this url to see if the queue sizes change
  6. If it works, push this to Heroku
  7. Enable the newly created worker manager on HireFire and it should work.

 

SPECIAL ALERT

If you don’t have a super specific case regarding history of your tasks, consider the setting


CELERY_RESULT_BACKEND = None

It will really, really save you tones of money and debugging time!

Final Thoughts

I wouldn’t recommend more, experimenting with heroku autoscaling and Hirefire.io! It is a decent stable solution and it works great with a mature framework as celery.

Apart from recommending our solution, this blog post is also a future reference for myself the next time I have a similar problem 😀 😀

 

 

Business Narratology: Telling a Story with APIs

This article will try to “sell” a different perspective for a story that you may be already familiar with. How good are you at telling a story?
Storytelling is an important skill for engineers and more specifically for us, the software developers. We often have to create something that our manager is not exactly sure what this is. Thus, it is our duty to create a workflow, some scenarios and finally agree about the desired outcome of the piece of software.
Engineers around the world have come up with so many different ways to develop scenarios for describing features. A new trend on this, is using tools like Trello or Aha! to create sketches of products and a basic roadmap on when to do what, etc etc etc.
In my understanding, APIs are all about telling stories. When you share an API, you share all the important blocks that a “writer” can reuse to create their own “stories”. Those stories can be useful business partnerships, hobby dev applications and much much more. No-one can actually limit the imagination of people who want to develop their story on top of your platform.
This is the kind of story that I will try and share here.
Let’s say you are just a guy with an awesome idea or  still a student at a university.
One day, you decide to get started with your cool idea and initiate the development of a  super recommendation engine on comic books.
You know soooooooo many cool facts and insights about comic books but you are not exactly sure how to write a recommendation algorithm and implement it.
  • If you are a Batman fan, you most probably will enjoy some Nightwing action.
  • If you like Superman, most certainly you are going to like Captain Marvel as well or some Greep Lantern Corps adventures.
  • If you hate Flash, you should read the “The Flash: Rebirth”, because everyone likes Flash. They just need to read the correct story!
You may think, that’s piece of cake. Hopefully there is Wikipedia!  You look there for popular recommendations algorithms. One thing links to another (this is how web actually works) and in the end you end up studying some super difficult algorithms to understand mathematics where an equation can cover the whole screen. There is nothing to laugh about it. Those things actually happen.
And they happened to me a lot…….. If you are brave enough, you may end up on some online lectures on Udemy, Udacity or Coursera.

But!!! There are so many cool ways to avoid such pitfalls. And we call them, APIs.
Last year for example, when I was in API Barcelona, on the second day of the conference I was fortunate enough to see so many solutions on the aforementioned problem that I could not even believe that was indeed true.
So, i decided to check for myself.
After five minutes of thinking, while i was walking next to my hotel (looking at the beautiful barceloneta, photo below),  i said that if I could manage to create the impossible, then I would be convinced that APIs actually work.
Barceloneta
The idea was “simple enough”. I wanted to retrieve all information from  ycombinator latest news and all comments from there, and only look to anything that is to any of my interested and people say good things about that. When something like this happened, i wanted to retrieve a new email.
  • And then i said, this is a super cool idea. Why not share it??
  • And what I did, i wanted to let others get notified as well!!!

So what do i need to succeed on such a goal?
  1. a way to retrieve articles and their comments
  2. identify what those articles are about and classify them to some popular entities and categories
  3. do some sentiment analysis based on the dialogue that was happening and the comments
  4. a way to send automatic emails on demand by my own domain (yes yes i wanted to use ycombinator-news@apilama.com)
To make the long story short, i had just a sticker from mailgun, i had drank a beer from mashape, alchemy API had given this day and extraordinary presentation on NLP APIs and said, why not…
CDXrMseWgAABeQG
I signed up to all three services and the first think I did back then, looked for some python scripts to test them.

The workflow was simple enough and started implementing it within minutes. Within an hour of coding, I had the first demo of my idea. Then, I was convinced!


This whole article is about you telling a story. And here I am telling a story about the “writer’s block” or more specifically about the developer’s block.
I wanted to experiment, I wanted to do something and when the idea hit me, I was able to implement it with just a few APIs.
APIs do just that. Help you by providing all the building blocks to make anything you could think of, a reality.
Going a step further,  APIs will describe the scenarios of future itself. It is funny how well modern sci-fi has reached a point of understanding APIs and using it in their stories.
maxresdefault
In a previous post of mine I tried to capture a bit of what I expect of the future and why APIs will play such an important role. As for today, simply put:
Each API is a collaborative tool that help businesses, wanna-become-businesses and hobbyists to share their information or consume others information in order to create relevant business models where each ones survival is dependent on the other ones surviving the business world.
In a few years from now, APIs will be everywhere and everyone will be talking WITH those. APIs will be intelligent and people should be educated to communicate and interact with them.

#APIs, #SmartCities, #IoT and more on @APIlama

Today I wanted to do some research on the topics that I like to post more about. Don’t get too alarmed, this is not an academic essay about the pros and cons of technology and stuff. I just “googled” the most hot topics of tech and the results can be found below.

To summarize this a little bit, those topics are:

Mainly I wanted to look out how those topics are discussed throughout the Internet and see if my personal gut is right on following after those areas.

It is pretty obvious from the graphic below that APIs, Web and Programming in general are pretty popular topics while Smart Cities and IoT only now start to pick up. Those were more or less what I was expecting to see. I was a little bit surprised with the peak of WWW the last year. This is something I did not see coming and I am not totally sure on why this happened.

I was also expecting to see more traffic on Smart Cities and IoT but maybe this means that Google needs more time to understand a new topic that people search and discuss about.

Screen Shot 2016-03-07 at 21.52.46

For this research I used Google Trends. This is a really nice tool to provide an overview of various topics across time and regions. You can find this specific research directly on Google Trends.

Regional Interest

Another area of interest apart from how those article perform over time, is how those perform over regions. Some of the results were surprising.

For example, I did not expect so many people from Sri Lanka or Tunisia to be interested on APIs. Mainly because I don’t know any developers over there 😛

Some other of the results were expected more or less one could say. For example it was no surprise that South Korea is so interested in IoT. Or Germany is searching a lot for World Wide Web, which is somewhat expected since they are trying to become the Silicon Valley of Europe.

What amazed me a lot, was the increased “googling” of Malta about Smart Cities. I did expect this for United Arab Emirates, Singapore and India who are considered pioneers on the area, but Malta was a surprise to me.With a little bit of searching I found out that Malta is deep on the Smart City game with their SmartCity project which btw, looks awesome!

Top Interests and Queries

Another interesting information that Trends show are the top topics regarding the overall category you are searching for and the most prominent queries.

I am only attaching the information about APIs, because the rest were not that interesting mostly cause either they are not that populated yet or they were not that relevant to a technical site.

Screen Shot 2016-03-07 at 21.53.52

Looking at the results, it totally makes sense that REST is there, and also JSON and Javascript. But Java???

46e039da6efd16c540b180b964d5a0278267ec0126da7e6f4fff0e3af0224b22

I am not a big fan of Java, but it is definitely not a term to attach with APIs. On the other hand, Java was the first to coin the term, but since then we’ve gone a loooong way.

In any case, it is definitely a good thing that people are looking for better programming paradigms and this is the whole thing about APIs.

APIS: Better programming, Better software quality

Create your Own Trend Graphics

This article is part of my research on the topics that I find “hot”. I would like to see what you also think about those trends and play with your own graphics.

Feel free to contact me with your ideas and I would love to share them in a next post!! As I have quoted many times in the past, this site is a collaborative effort and my posts are only a trigger for further discussion.