Download now.

Changes

Fixes

• Issue 63: the REST API endpoint for the setItemTag has been corrected. See Upgrade Notes.
• Issue 64: Values of λ that are too large should cause the model build to be rejected, rather than cause runtime errors related to large user and item vectors during fold-in.
• Issue 65: Evaluation framework can again handle files with tag data
• Issue 66: Resolved an issue that generated an exception on adding new data, when few or no clusters are present, but clustering is enabled
• Issue 67: REST API endpoints correctly return a client error instead of exception when expecting a URL path but none is presented
• Issue 68: Fixed error in /estimateForAnonymous endpiont that returned a server error / exception when all items in the request were unknown, instead of a simple client error
• Issue 70: Mean average precision computation has been corrected
• Issue 74: Computation Layer: Fixed a possible incorrect cluster assignment; can be incorrect when input is small, cluster count is high
• Issue 75: Fixed an important error in the Computation Layer (only) that can cause recommendations or similar items, or in some cases steps in iteration, to be missed in some cases where multi-threaded reducers are used.

Other

• Issue 72: When number of feature changes, still bootstrap model state from previous generation's Y matrix vectors
• Issue 77: Standardize output of AllItemSimilarities, AllRecommendations and direct output to a file

Upgrade Notes

Serving Layer

• The endpoint for setItemTag was declared incorrectly. Requests should access /tag/item/[itemID]/[tag], not /tag/item/[tag]/[itemID]. Callers who access the REST API directly will need to use the corrected path.
• Stand-alone: the format of model.bin.gz has changed in a backwards-incompatible way. Remove it and let the server build a new model.
• Stand-alone: AllItemSimilarities and AllRecommendations programs now require an --outFile parameter instead of printing to stdout. Their output format has changed as well, to match the output of the Comptuation Layer. See the javadoc.

Sean will be speaking again at Berlin Buzzwords this year, on behalf of Myrrix. The topic?

The talk will explore some real world examples of dealing with scale by doing much less work but still getting useful answers. More details to come.

Buzzwords is a strong Big Data conference in Europe and even stronger this year. Early tweets from other speakers suggest a great slate of talks are shortly to be announced. Grab a ticket for Berlin Buzzwords 2013 while early-bird pricing remains in effect.

• New "tag" API which allows callers to record interactions between users/items and concepts, or labels, which inform the model but are not returned in results. For example a user can be "female" and an item "yellow".
• New estimateToAnonymous method, which functions like estimatePreference for "anonymous" users, like recommendToAnonymous.

Fixes

• Minor fixes and improvements to model loading and persistence in Serving Layer.
• Java client can now fail over to replicas in more error cases, like mid-request interruption

Other

• Improved speed in Computation Layer.
• IDRescorer behavior has slightly changed for recommendToMany. See Upgrade Notes.
• Default value of model.als.alpha and negative strength values has changed. This should result in a small improvement in quality on many data sets. See Upgrade Notes.
• Computation Layer model.cluster.k parameter has been split into model.cluster.user.k and model.cluster.item.k to control k for user and item clustering, respectively.
• The Computation Layer will now delete old generations, keeping 10 generations by default. If a different value is desired, set model.generations.maxToKeep.

Upgrade Notes

All

• The default value of model.als.alpha has changed from 40 to 1. This affects results if the model build process (Computation Layer or stand-alone Serving Layer) does not specify a fixed value for this system property. The new default should result in an improvement in quality for many data sets.
• Handling of user-item pairs with negative strength values has changed. This has no effect for data sets that contain no negative strength values. It should have a small and generally positive effect on other data sets.

Serving Layer

• The behavior of IDRescorer has slightly changed for recommendToMany. The function is applied once per item, not for every user and for every item. That is, it is applied after the average over user scores has been taken, not before.

Computation Layer

• Computation Layer model.cluster.k parameter has been split into model.cluster.user.k and model.cluster.item.k to control k for user and item clustering, respectively. If using the old parameter, specify one (or both) new paramters now instead.
• Note that the Computation Layer will now delete old generations, keeping 10 generations by default

Myrrix and NGDATA have worked together to integrate Myrrix's products as a component of Lily's real-time recommendation engine. Myrrix's collaborative filtering, as part of Lily's machine learning, enables sophisticated contextual and personalized product recommendations. Enterprises can predict an individual's preferences toward a product or service by observing group behavior such as purchase history, brand affinity, social interaction, POS transactions, web clickstreams and geo-location information to deliver highly accurate recommendations.

About NGDATA

NGDATA™ is the customer intelligence management solutions company that empowers enterprises seeking greater customer lifetime value to drive continuous, actionable insights to enable sales and increase customer loyalty. The company does this through its unique combination of Big Data management and machine learning technologies in a single integrated solution. Recently named one of Bank Systems and Technology Magazine's "Top 7 Big Data Players to Watch," NGDATA is headquartered in Ghent, Belgium with offices in New York City and San Francisco. The company provides solutions to data-driven sectors such as financial services, retail and media/publishing. Please visit www.ngdata.com for more information or follow on Twitter @ngdata_com or connect with the company on LinkedIn.

The distributed Myrrix Computation Layer runs on all recent stable Apache Hadoop releases, and compatible distributions of it. Because the Hortonworks Data Platform follows Apache releases closely, Myrrix is already comptabile with the Hortonworks distribution, from version 1.0 to the latest 1.2.1 and beyond. The partnership will help ensure ongoing technical compatibility and optimal operation for customers.

With Myrrix, Hortonworks customers can leverage their Hadoop cluster to build real-time, scalable clustering and recommender systems. This "Big Learning" can be applied to click-stream data, purchases, shares, social links, or any association data. These machine learning models can be integrated into real-time production systems and increase conversion rates, drive customer engagement and content consumption, target ads, discover user segments, and more.

Myrrix is also a qualified services provider, helping customers design, build and integrate world-class Big Learning systems.

Read more about the Hortonworks Data Platform and the Myrrix Computation Layer.

• Computation Layer now supports clustering with kmeans++ / spectral clustering
• New similarityToItem method to retrieve individual item-item similarities
• New mostPopularItems method to list items associated to the largest number of users
• New /user/clusters/* and /item/clusters/* methods to query clusters
• New "client thread" in the Serving Layer can be used to pull/poll for data from an external source and interact directly with the Serving Layer
• Optional basic DoS attack protection in the Serving Layer

Other

• RescorerProvider can now use the local recommender instance in its operation
• More support for multiple RescorerProvider classes at once
• Model building dynamically calculates number of iterations needed; default biases to more initial iterations (and better accuracy) than previous default.

Fixes

• Resolved an issue that could cause two Computation Layer computations sharing the same bucket to interfere. Support files like keystore.ks and rescorer.jar, which are normally placed in sys/, should now be placed in instance/sys/.

Upgrade Notes

All

• The model.iterations parameter no longer has effect; iteration count is no longer explicitly specified, but instead determined dynamically by a convergence criterion: the average absolute difference in estimates for user-item pairs between iterations. The model.als.iterations.convergenceThreshold parameter, which defaults to 0.001, will generally cause significantly more iterations to be run. Increase (to 0.01 for example) for fewer iterations. The number of iterations can be capped by setting model.iterations.max to the (positive) max number of iterations to run.

Computation Layer

• An issue has been resolved that could cause two Computation Layer computations sharing the same bucket to interfere. As part of this, support files that were previously bucket-specific, and were previously placed in sys/ under a bucket, like keystore.ks, clientthread.jar and rescorer.jar, are now instance-specific. They should now be placed in instance/sys/. Stop the Computation Layer instance(s), and for each bucket, copy its sys/ directory into each instance/ subdirectory. Restart the Comptutation Layer(s). Note that the file sys/myrrix.jar, will be regenerated. No migration is needed if none of the files listed above are used.

Read the Nokta Case Sketch to see how their video sharing site, İzlesene.com, added 600,000 views per day by easily incorporating a scalable recommender system from Myrrix.

• Serving Layer can now run in read-only mode to serve a model but not accept updates
• /ingest endpoint can now accept web-browser-style file uploads
• New unified license system is in place. Obtain a trial license from myrrix.com

Fixes

• Standalone Serving Layer .war file deployment works again
• HTTP DIGEST authentication is now made compatible with all clients and the latest Tomcat again
• Choice of partition and replica is now deterministic in the client. This avoids problems where two replicas presented slightly different results to the client on successive requests.
• Removed "_LOCK" file mechanism in Computation Layer in favor of querying the live cluster state
• Stand-alone serving layer avoids a possible exception when recomputing a new model on new items
• Several fixes for problems in computing models over very small input
• Repeated updates from new users and items should not be able to produce extreme values in the model now even after many updates
• Java client library now supports rescorer parameters
• Added missing recommendToMany to client CLI, changed some APIs to be more consistent, and fixed other small issues in the CLI.

Other

• Faster display of self-organizing map visualization for moderate to large data sets.
• Client command line has changed slightly, to take optional argument like "howMany" as a flag (e.g. --howMany 5 instead of stand-alone argument)
• mostSimilarItems and recommendToAnonymous will return a result if at least one argument item exists, rather than if all exist
• Update to support Hadoop 1.1.1 and latest Amazon EMR releases

... This post details how Myrrix processed 337 million Wikipedia links, consuming 1,320 instance-hours (normalized to m1.small) over 10 hours, for under $44 (a savings of 67% versus On Demand).