An experiment on the weakness of reputation algorithms used in professional social networks: the case of Naymz

by Marco Lazzari

Abstract

This paper is part of an ongoing research on web reputation, and presents an evaluation of the method used by the professional social network site Naymz to assess the online reputation of its members. By creating six ad-hoc professional communities with different profiles of users and network topologies, it was found that the highest level of online reputation can be easily achieved by engaging a small social group of nine persons or less who connect with each other, providing positive feedbacks and reciprocal endorsements.

Introduction

The sudden world-wide spread of social networks has transformed them into a rich source of personal information about their members. Personal profiles from social networks are nowadays commonly used by companies looking for new partnerships or employees or by universities selecting freshman applicants (Williams and Lounsbury Morrow, 2008). Therefore, whilst structural properties of social networks have been broadly investigated (Knoke and Kuklinski, 1982; Wassermann and Faust, 1994; Scott, 2000; Knoke and Yang, 2008), the dual problem of building and assessing the web reputation is assuming ever more importance, and has been receiving growing attention by scholars (Goldbeck and Hendler, 2004; Jøsang, A. et al, 2007). In this paper, we deal with it with reference to the professional networks domain.

Professional networks, such as LinkedIn, Xing, Naymz, are web communities devoted to interactions and connections of a formal business nature, rather than informal social exchanges, which is the case of social networks such as Facebook or Netlog, which are exploited by their users for fun, recreation, pleasure, discussion or for organizing and coordinating groups and events.

Professionals can exploit professional networks to connect to current or former colleagues, establish new partnerships, communicate effectively and discover new professional opportunities. Basic services provided by professional networking platforms consist in functions to describe the member's profile (personal data, education, professional history, professional and personal web resources) and to connect to other members for building personal small worlds (Milgram, 1967; Travers and Milgram, 1969; Schnettler, 2009).

These networks often encourage users to endorse other members, so that the online reputation of a member, fundamentally based upon his/her own profile, can be enforced by the judgment that contacts express through the endorsement.

The professional network Naymz sets itself apart from competitors, because it provides an online algorithm to evaluate the reputation of its members: each time a member establishes a new connection, he/she is asked if he thinks that his new contact is honest, if he would recommend him for jobs, if he would like to be worked with, if he wants to be considered as a reference for the new contact and if he wants to endorse him. These answers increment a RepScore, that is the reputation score of the member, on the ground of an algorithm that assigns points to each profile and to each answer, reference, and endorsement got from contacts, the influence of a member on another one being proportional to his own RepScore. The RepScore is then mapped onto a scale from 1 to 10 RepScore levels: thresholds from a level to another one are constantly changing and are dynamically calculated on the ground of the average scores of the Naymz users (Naymz claims to have more than 1 million members).

Naymz's algorithm is therefore an attempt to develop a strategy of people ranking similar to the page ranking model commonly used by search engines to measure the relative importance of each element of a set of links which satisfy a query.

Data and methods

In the framework of an ongoing study on web reputation and trustworthiness of web communications, we performed an experiment to reveal some elements of weakness of Naymz's people ranking algorithm; our hypothesis was that the algorithm is weak in the sense that a low number of motivated cooperating people can easily get the maximum RepScore level.

In order to verify our hypothesis, we created six virtual professional communities, each of which was centered on a profile which was directly managed by the principal investigator: in each community, this main node was connected to all the other members.

Three of the six communities, which we shall call in the following real communities, were made of students of the same class, that is 9 college students, 16 graduate students of a course on Educational technology and 36 graduate students of a postgraduate school for teacher education, who were free to contact other members of the same class group. Two other communities, which we shall call in the following virtual communities, were composed of respectively 9 and 15 members, whose profiles were managed by some research assistants, according to a specific policy related to the network topology. The sixth community, which we shall call in the following hyper-real community, was built by connecting the main node to other members of Naymz, not known in the real life by the investigators, who had already reached the highest RepScore level.

Both in the real and in the virtual communities, the topology of the networks and the strength of the connections were different, on the ground of the different groups and the different commitment they had in the research; with reference to the virtual groups, we experimented two topologies, a balanced binary tree and a complete graph.

In each network the initial task for each member was the creation of the profile: students were free to provide rich or basic biographies; we have noticed that (within our groups) the richness of the profile is inversely proportional to the age of the participant.

Then, each member was held to contact the main node of the network; eventually, students were free to contact other members of the network and to exchange references and endorsements. After some weeks, we noticed that many students had received friendship requests from people from all over the world: with our graduate students we decided to ignore such requests, while undergraduates, who were loosely and indirectly connected with the research team, independently decided to accept them. As far as the virtual communities, they were tightly coupled, in the sense that each member positively answered to all the questions of the contact procedure for each of its contacts, as well as referenced and endorsed all of them.

Findings

As a result of our activities, in four cases out of six the main node of the network easily reached the highest RepScore level (that is 10). In the larger community (36 students), the other members were loosely interconnected and reached lower levels (average RepScore: 142.7, standard deviation: 110.4; avg RepScore level: 5.7, std dev: 1.8), while the main node was pushed up by the high number of contacts (top RepScore: 907): despite the individual scores were low, the composition of their contribution as evaluated by the ranking algorithm was enough to bring the node to the maximum level.

In the case of the network related to the course on Educational technology, the students' commitment was higher than in the previous case, the net resulted highly interconnected and almost all members reached the top (avg RepScore: 659.3, std dev: 225.4; avg RepScore level: 9.4, std dev: 1.1).

The sole case of RepScore lower than 10 for the main node is given by the college students' network, where the main node did not get the top (RepScore level: 8), while 4 out of 9 other members did (avg RepScore: 529.1, std dev: 499.6; avg RepScore level: 7.3, std dev: 2.8): the reason was that, as we said above, those students accepted friendship requests from outside the school and even contacted external members with maximum reputation level for inviting them to join their personal network, in order to increase their own RepScore. In fact, this behavior is coherent with what Naymz states for describing its distinguishing features: "With Naymz, you can […] find and connect with anyone on the network - including people you know and don't know", even if it appears a clear contradiction of the principle of online reputation based on endorsements made by persons who know your qualities. We have spotted that the phenomenon of the "false friends" is rather common: as soon as Naymz's users get high RepScores, they receive contact requests by other members, "fortune-hunters" who are simply looking for the bonus points that a new contact may supply. Most of them ask for endorsements as well, and provide unsolicited endorsements, which are often recognizable by their vagueness (for instance: "Terrific networker. Recommend.", "Thanks for giving me the great opportunity to network with you", or even "Hi, can you endorse me?").

This community showed that (i) 4 top-score contacts are not enough for a member to get the maximum reputation level; (ii) 9 connections are not enough, if the network is loosely connected (the third community has a low number of references and endorsements); (iii) "friends of friends" do not seem to have direct impact on RepScore.

On the other hand, the second community pointed out that (iv) less than 16 well connected members may reach the highest reputation level (we say "less than", because some of the members of that community were not really active and probably their impact on the rest of the net would had been achieved by substituting them by a lower number of more active members).

For these reasons, in order to better define a limit value for the number of nodes necessary to get the best results, we decided to build the other three networks.

The hyper-real community was built by contacting reputable professionals chosen among those who had already reached the top RepScore level and we quickly succeeded in getting the same result, even if no one of them did know our main node in real life (nobody rejected our contact request). The reputation level of 9 was achieved after 6 days, with 7 contacts; the top level was reached after 11 days, with 11 contacts. It should be noticed that free accounts do not allow to have more than 10 friendship requests waiting for an answer, and this is a limit to the speed of growth of the network. Our main node referenced and endorsed all the contacts, but was endorsed only by three of them. This means that the value of 11 is still overestimating the cardinality which is enough to get the maximum RepScore level.

As far as the other two virtual networks, the root of the balanced tree reached level 7, while most nodes got 8 (the root is connected to its two descendants, while each non-terminal node is connected to two descendants and an antecedent) ; and all the nodes of the complete graph reached the top: the former result enforces the assumption (iii), whilst the latter result demonstrates that a group made of 9 people who agree to cooperate and endorse each other can get the maximum RepScore for each member of the group.

We also lay stress on the fact that we sketched minimal profiles for the members of the virtual communities: should the members provide richer profiles, which enhance the score, less than 9 members would be enough.

A monitoring of RepScore evolution showed that "bridges" between networks are not necessarily high reputation members, while "hubs" (Barabási, 2002), that is members with many connections, are usually at the top level and heavily influence their contacts. When bridges contact hubs, their scores are significantly higher: Table 1 shows the RepScore levels / RepScores of four bridges we have established among our six communities; they differs by two parameters: (1) contact to main node / to weak node; (2) strong tie (reciprocal reference and endorsement) / weak tie.

Table 1. RepScore levels / RepScores for bridges

	Bridge among main nodes	Bridge among weak nodes
Strong ties	9 / 475	8 / 371
Weak ties	7 / 200	7 / 161

Conclusions

Our experiment shows that the algorithm currently used by the professional network Naymz to evaluate the online reputation of its members is rather weak against two kinds of malicious behavior: (i) the "class action" of a small group of fellows (nine and even less could be enough), who set up their own sub-network of friendship connections, references and endorsements, (ii) the activity of a single member, who contacts other members randomly, or more often by selecting them on the ground of their reputation, since the friendship of a high-reputation member has a stronger impact on the evaluation of the reputation score.

In fact, the basic problem is not the algorithm by itself, but the lack of quality of the data used in the processing: both the user's identity and the profile data are subjective and not verifiable (Naymz assigns extra points to identities verified by Trufina, but even verified users can add false information about themselves, in order to get more points); there are no barriers to avoid false friendships; courtesy endorsements are common.

Our future work aims at understanding if these problems are generalized to other professional social networks and if there are ways to solve them.

Acknowledgements

I wish to thank Andrea Mazza and Maurizia Calabrese, who helped me monitor some of the networks; my students of the SILSIS (Scuola Interuniversitaria Lombarda di Specializzazione per l'Insegnamento Secondario) of Bergamo e Brescia, those of the course on Educational technology at the Faculty of Educational Studies of the University of Bergamo, and professor Paolo Macchi and his students of the ISIS "C. Facchinetti" of Busto Arsizio for the time they spent to help me develop the network communities used in my experiment.

References

Barabási, A.-L., 2002. Linked: the New Science of Networks. Perseus, Cambridge, MA, USA.

Goldbeck, J. and Hendler, J., 2004. Accuracy of Metrics for Inferring Trust and Reputation in Semantic Web-based Social Networks. Proceedings of the 14th International Conference on Knowledge Engineering and Knowledge Management (EKAW 2004). Whittlebury Hall, UK, pp. 116-131.

Jøsang, A. et al, 2007. A Survey of Trust and Reputation Systems for Online Service Provision. Decision Support Systems, Vol. 43, No. 2, pp. 618-644.

Knoke, D. and Kuklinski, J.H., 1982. Network Analysis. Sage Publications, Newbury Park, CA, USA.

Knoke, D. and Yang, S., 2008. Social Network Analysis. Sage Publications, Thousand Oaks, CA, USA.

Milgram, S., 1967. The Small World Problem. Psycology today, Vol. 1, No. 1, pp. 61-67.

Schnettler, S., 2009, A structured overview of 50 years of small-world research. Social Networks,Vol. 31, No. 3, pp. 165-178.

Scott, J., 2000. Social Network Analysis. Sage Publications, Thousand Oaks, CA, USA.

Travers; J. and Milgram, S., 1969. An Experimental Study of the Small World Problem. Sociometry, Vol. 32, No. 4., pp. 425-443.

Wassermann, S. and Faust, K., 1994. Social Network Analysis. Cambridge University Press, Cambridge, UK.

Williams T.D. and Lounsbury Morrow, A., 2008. Want To Know Your Employees Better? Log on to a Social Network. The Alabama Lawyer. Vol. 69, No. 2, pp. 131-135.