1. FRAMEWORK AND SCORING

   1. Our safety methodology is built on the following four main pillars: regulation, consumer protection, market fairness and transparency.
   2. Each of these pillars contains four sub-categories, to give 16 sub-categories in total.
   3. Each sub-category consists of several qualitative and/or quantitative factors trying to grasp key aspects of each such sub-category.
   4. Each sub-category is then divided further into three different TIERs and crypto exchanges are ranked and assigned to a particular TIER.
   5. Best performing crypto exchanges were assigned to TIER 1 and given 5 points out of 5, crypto exchanges with ‘average’ results were assigned to TIER 2 and given 3 points out of 5 and worst-performing crypto exchanges were assigned to TIER 3 and given 1 point out of 5. 
   6. Given that assessments here might involve judgment calls and deal with elements that are hard to quantify, there is a subjective overwrite mechanism built into the methodology allowing one extra point either to be added to or deducted from an exchange falling between two TIERs.
   7. Points received based on TIER rankings for each sub-category within each pillar are added together and divided by four (the number of sub-categories within each pillar) resulting in an average score (on a scale of 1 to 5) for each pillar.
   8. Then such average scores are also added together and divided by four again (the number of pillars), given that (like sub-categories) pillars are also equally weighted.
   9. As a result, we get a comparable final safety score for every exchange.

2. DESCRIPTION

1. REGULATION PILLAR

1. Jurisdiction: an internal list of 50 jurisdictions was created by using indices published by the World Justice Project, the World Bank, the United Nations and Transparency International (all of them focusing on different aspects of the countries’ regulatory environment and justice system). Crypto exchanges established and/or operating from a country within the top 25 countries got a TIER 1 designation, crypto exchanges established and/or operating from a country falling between the 26th and 50th on our list went to TIER 2 and crypto exchanges established and/or operating from a country not included in our list went to TIER 3.

2. License: reference point of the assessment here is the EU regulatory regime. Deemed equivalency might be established when an entity established and/or operating within a TIER 1 or TIER 2 jurisdiction is also subject to authorisation, supervision (conduct, prudential etc.) corresponding to the relevant EU regulatory regime and is authorised to perform relevant investment and ancillary services a defined under the EU regulatory regime. TIER 1 category here consists of crypto exchanges that are holding a MiFID II or equivalent license, TIER 2 category includes crypto exchanges operating as payment service providers or e-money issuers holding a PSD2 or equivalent license and unregulated entities and entities where there was no information found went to TIER 3.

3. Regulatory incidents: mean either ongoing investigations launched by a government agency, authority or other official body having jurisdiction over the crypto exchange and/or its owners, directors, officers, employees etc. or any regulatory fine or sanction imposed on a crypto exchange and/or its owners, directors, officers, employees etc. by such entities. TIER 1 category here consists of crypto exchanges where there were no ongoing investigation(s) and/or regulatory fines, other sanctions in the last 5 years, TIER 2 category includes crypto exchanges where there were no ongoing investigation(s) but regulatory fine(s), other sanction(s) with minor implications were imposed in the last 5 years and crypto exchanges where there were either ongoing investigation(s) or regulatory fine(s), other sanction(s) with major implications in the last 5 years went to TIER 3. Differentiation between minor and major implications is to be made on a reasonable basis taking into account all the relevant circumstances and available information.

4. Membership in self-regulatory organizations (SROs): because there is no uniform regulatory regime for the crypto industry, SROs might play an important role in shaping future regulatory agendas by participating in discussions with regulators and policymakers. Hence we decided to include them in our methodology. We have selected nine SROs based on a number of criteria including number of members; their legal status; composition of their board/leadership and their documented involvement in rulemaking and/or policy discussions about regulation concerning the crypto ecosystem. TIER 1 category here consists of crypto exchanges with memberships in multiple self-regulatory organisations, TIER 2 category includes crypto exchanges with membership at least in one self-regulatory organisation and crypto exchanges with no membership or no public information about membership(s) went to TIER 3.

2. CONSUMER PROTECTION PILLAR

1. Protection for crypto holdings: as traditional deposit insurance schemes are not covering crypto assets, commercial, captive or other self-made insurance solutions are trying to fill the gap. TIER 1 category here consists of crypto exchanges holding commercial and/or captive insurance policies, TIER 2 category includes crypto exchanges operating with self-made insurance solutions (i.e. internal consumer protection funds being basically a pool of certain segregated crypto assets maintained and controlled by such exchanges on the blockchain) and crypto exchanges where either no credible information available or no insurance is being provided went to TIER 3.

This sub-category is not relevant at pure derivatives exchanges like Kraken Futures or FTX US Derivatives where either no or fairly minimal crypto asset is actually being held (as margin) on behalf of customers hence such entities automatically received TIER 1 categorization given that consumer protection issues resulting from the lack or inadequacy of the available protection do not arise in their case.

2. Protection for fiat holdings: this factor is measuring the level of consumer protection for fiat assets held at a given crypto exchange on behalf of customers (but also providing useful information about banking relationships of such crypto exchanges). TIER 1 category here consists of crypto exchanges where there is (pass-through) deposit insurance (meaning that the exchange is holding client money segregated from its own at banks participating in deposit insurance schemes), TIER 2 category includes crypto exchanges where deposit insurance is not available and it is clearly communicated to customers and entities where no credible information found went to TIER 3.

3. Hacking incidents: the term ‘hacking incident’ covers any successful attack against the systems of a crypto exchange whether it resulted in a monetary loss, loss of crypto-assets and/or loss of customer data or other data leakage. We have compiled publicly available data about hacking incidents going back to the Mt. Gox hack in 2014 and calculated the average dollar amount of the losses resulting from these hacks for each year and then for the entire period between 2014 and April 2022. When calculating the average, we have assigned zero value to hacks which resulted “only” in data loss/data leakage or where the lost amount was not disclosed. The loss which is either equal to or greater than 50% of such average amount (2014-2022) will be taken as an incident having a major implication and anything less than this threshold will be taken as an incident having a minor implication. TIER 1 category encompasses crypto exchanges where there were no incident and/or no successful hacking attack in the last 5 years, TIER 2 category includes crypto exchanges suffered a hacking incident with minor implications in the last 5 years and entities suffered either a hacking incident with major implications or multiple hacking attacks in the last 5 years went to TIER 3.

4. Operational resiliency: covering the following aspects: (i) having a trade surveillance system in place (to monitor trading activity and fight market manipulation), (ii) having a bug bounty programs or penetration testing (to continuously test safety and resiliency) (iii) having third party security assessment and certification (to obtain verification from a credible and independent source) and (iv) having a business continuity/disaster recovery plan. TIER 1 category encompasses crypto exchanges having four out of four elements in place, TIER 2 category includes crypto exchanges having three out of four elements in place and entities with two or fewer elements in place went to TIER 3.

3. MARKET FAIRNESS PILLAR

1. Restrictions on exchange proprietary trading: in case crypto exchanges engage in proprietary trading on their own venue this means that a customer’s order to buy or sell crypto could have been filled not by another customer, but by a “trading desk” run by the crypto exchange itself, trading on behalf of the crypto exchange for its own account. As compared to the traditional financial sector there is much less guarantee to ensure that such trading desks do not have an informational advantage over customers. This risk does often materialize within contractual terms when allowing such trading for profit to happen. However, it is possible that crypto exchanges might act as market makers submitting both buy and sell orders for the same assets in order to promote liquidity. Such activity is common in the traditional securities marketplace, particularly in broker-operated alternative trading systems but it requires a significant commitment to customer protections and transparency to remain in compliance with applicable laws and there is way less transparency in the crypto sphere. In addition, when a significant percentage of the volume in one or more crypto assets on a venue is attributable to one source, customers face the risk that the availability of liquidity in those assets could change, without notice and at any time, including when liquidity is needed most – namely, in times of market turmoil which calls into question whether the market for crypto on those platforms is as robust as customers might believe it to be. TIER 1 category encompasses crypto exchanges where clear and detailed information / confirmation about such restrictions can be found on the exchange’s website, TIER 2 category includes crypto exchanges where some information is publicly available outside the website but not enough to have a satisfactory answer as whether such practice is prohibited or not and entities where either no restriction is imposed or no credible information available at all about such restrictions went to TIER 3.

2. Restrictions on employee trading: another feature that distinguishes crypto exchanges from traditional securities markets is that owners and employees can trade directly on their own platforms which is another potential source of conflict of interest. TIER 1 category encompasses crypto exchanges where clear and detailed information/confirmation about restrictions on employee trading can be found on the exchange’s website or can be obtained from a credible source, TIER 2 category includes crypto exchanges where some information is publicly available outside the website but not enough to have a satisfactory answer whether there is a restriction imposed on employee trading or not and entities where either no restriction is imposed or no credible information available at all about such restrictions went to TIER 3.

3. Coin listing criteria/asset framework: in the majority of the cases crypto exchanges are compensated for listing crypto assets. As of today, there are no regulatory standards for determining whether a particular virtual asset can or should be listed on a trading platform, it is important to have a look at what internal rules are governing this activity at crypto exchanges. TIER 1 category encompasses crypto exchanges having clear and detailed information/methodology on the website either as a standalone document or part of standard documents, TIER 2 category includes crypto exchanges having some information on the website but not enough to have a satisfactory overview/understanding about the listing rules/asset framework and entities where either no information or no credible information available went to TIER 3.

This sub-category is not relevant at pure derivatives exchanges such as Kraken Futures or FTX US Derivatives where no coin is listed hence such entities automatically received TIER 1 categorization given that issues resulting from the lack of or a non-transparent coin listing criteria/asset framework do not arise in their case.

4. Exchange issued utility tokens and ‘shady’ stablecoins: utility tokens carry a potential conflict of interest issues resulting from their different functionalities as they can be (i) ‘gas tokens’ fuelling transactions within different ecosystems, (ii) ‘crypto flavoured loyalty tokens’ granting discounts on trading fees and (iii) instruments you can trade like altcoins. However, the vast majority of such utility tokens only imitate crypto as they are issued entirely privately, traded on different exchanges and the issuer crypto exchanges control almost everything about them and they tend to work however issuers need them to at the time. Large portions of such utility tokens are owned by the issuers’ founding team (increasing the risk of insider trading) and some of these tokens are also used to back internal consumer protection funds. ‘Shady stablecoins’ mean stablecoins where either no or minimal credible information is available about the assets backing them (for example available auditor attestations about underlying assets do not seem to be published on a regular (i.e. monthly) but rather on an ad hoc basis, accounting firms providing them changed several times in the last couple of years and/or there are also proven cases when their issuers made false representations about the availability of underlying assets. TIER 1 category encompasses crypto exchanges where neither utility token nor stablecoin is issued by the crypto exchange (and/or its affiliate) and listed on the crypto exchange, TIER 2 category includes crypto exchanges where only own stablecoin (with credible information about the assets backing such coin) is issued by the crypto exchange (and/or its affiliate) and also listed on the crypto exchange and entities where either own utility token or shady stablecoin is issued by the crypto exchange (and/or its affiliate) and also listed on the crypto exchange went to TIER 3.

This sub-category is not relevant at pure derivatives exchanges such as Kraken Futures or FTX US Derivatives where no utility token and/or shady stablecoin is listed hence these entities automatically received TIER 1 categorization given that issues resulting from the issuance and/or use of their own utility token and/or shady stablecoin do not arise in their case. As part of the subjective overwrite mechanism where there is a possibility for derivative products to be either settled and/or margined in shady stablecoins, one point was deducted.

4. TRANSPARENCY PILLAR

   1. Corporate transparency: covering the availability of sufficient information about a crypto exchange’s (i) foundation date, (ii) corporate structure, (iii) ownership structure and (iv) senior management on such crypto exchange’s own website. TIER 1 category here consists of crypto exchanges where 3 out of 4 of information is publicly available on the exchange’s website, TIER 2 category includes crypto exchanges where 2 out of 4 of information is publicly available on the exchange’s website and entities where 2 or less information out of 4 is publicly available on the exchange’s website went to TIER 3.

   2. Financial transparency: means the availability of a crypto exchange’s audited financial statements and/or annual or half-yearly reports and/or other equivalent financial information. TIER 1 category here consists of crypto exchanges where these documents are publicly available on the exchange’s website, TIER 2 category includes crypto exchanges where financial information is published on the exchange’s website but is not audited and entities where these documents are not available went to TIER 3.

   3. Product transparency: looks at the product spectrum offered by crypto exchanges and follows ‘the simpler the better’ approach by relying on empirical evidence about retail investors’ vulnerability to significant decision making weaknesses and biases, having a limited ability to monitor risks. Keeping in mind that this is the case in the traditional financial sector then it can be assumed that the problem is all the more prevalent within the crypto ecosystem which is way less regulated and continuously makes available complex and largely unregulated products. There is a sharp contrast here with the sophisticated product oversight regime (e.g. under MiFID II) which used to cover and regulate the whole product life-cycle along with liability aspects. TIER 1 category here consists of crypto exchanges with simple and relatively easily understandable product offerings (mostly buying and selling crypto on the spot market), TIER 2 category includes crypto exchanges with intermediate product complexity (e.g. spot crypto and other relatively simple products like simple staking, earning and/or properly regulated tokenized traditional financial instruments (for example tokenized shares) and entities with complex product offering (e.g. spot crypto and various crypto derivatives, leveraged tokens, earning, staking, liquidity farming, crypto loans, crypto savings account, NFTs etc.) went to TIER 3.

4. Legal transparency: meaning the availability of (i) a detailed AML/KYC policy, (ii) detailed privacy policy, (iii) key customer contract(s) – on the exchange’s website and also (iv) the legal assessment of whether contractual wording of such key customer contract meets our internal minimum criteria. TIER 1 category here consists of crypto exchanges where all documents are available on the website and key customer contract(s) meet our internal minimum criteria, TIER 2 category includes crypto exchanges where either privacy or AML policy is missing and other key customer contract(s) meets our internal minimum criteria and entities where key customer(s) contract is not meeting our internal minimum criteria (irrespective of the availability of other documents) went to TIER 3. Please note that using the subjective overwrite mechanism 1 point might be awarded to a crypto exchange in case all the documents are available but the key customer contract does not meet internal minimum criteria by missing one element of the legal assessment outlined below.

Our internal minimum criteria looks into whether key customer contract(s) contain clear and detailed provisions about (i) contracting legal entity acting on behalf of the crypto exchange as well as the capacity in which is acting, (ii) governing law (must be the law of a TIER 1 or TIER 2 jurisdiction), (iii) dispute resolution where arbitration is not the only option for the customer and (iv) terms of the key customer contract must be fair and transparent (the legal basis for transparency and fairness assessment is the UK Consumer Rights Act 2015 (and associated CMA guidance).