Methodology



Scoring Methodology

The following document outlines the scoring system for each device’s component. The weights given to each component and their properties are provided based on the threat model discussed next. Additionally, the scoring system can be adapted for any deployment environment and threat model by changing the weight distribution around the components’ properties. Interested parties can find a copy of an evaluation template here.

  1. Overview and Terminology
  2. Threat Model
    1. Type of Attackers
    2. Examples
  3. Scoring Rubric
    1. Device
    2. Mobile Application
    3. Cloud Endpoints
    4. Network Communication
  4. Score Calculation
    1. Example

Overview and Terminology

Smart-home devices have four main components:

  1. The device - the hardware purchased (Alexa, SmartThings, Sonos, etc.).

  2. The mobile application - the companion mobile application that interacts with the device (Android or iOS application.)

  3. Cloud endpoints - Internet services that the device or the mobile application communicate with.

  4. Network communication - Network traffic between each component (local and Internet traffic).

Each of these components has properties that are used as features to compute a score.

Device properties

The device properties the following:

  1. Internet Pairing - The configuration of network credentials to connect the device to the Internet.

  2. Configuration - The device configuration during setup phase, creating an account, setting up preferences, etc..

  3. Upgradability - The device’s upgrade options. Does the device update automatically or requires user’s interaction?

  4. Exposed services - Visible running services on the device, like UPnP, mDNS, HTTP server, etc..

  5. Vulnerabilities - Running services on the device that contain vulnerabilities, which are scored based on CVSS.

Mobile Application properties

The mobile application properties are based on static analysis to identify three types of security issues.

  1. Sensitive Data - Sensitive data includes artifacts like API keys, passwords, and cryptographic keys that are hard-coded into the application.

  2. Programming Issues - Implementation errors and incorrect use of libraries include weak initialization vectors in cryptographic functions or guessable seeds to pseudorandom number generators.

  3. Over-privileged - Mobile applications request excess permissions that are not required or used in the application code.

Cloud endpoints

The cloud endpoint properties are based on the assessment of services that the device and/or the mobile application communicate with. There are three properties to inspect:

  1. Domain categories - Domain categories defines three main categories, namely first-party, third-party, and hybrid. First-party domains are endpoints that are owned and managed by the vendor of the product. Third-party domains are endpoints that use external services like Google Maps. Hybrid domains are endpoints that are run on cloud infrastructure like Amazon or Azure, but managed by the vendor of the device.

  2. TLS configuration - TLS configuration refers to the proper setup of TLS/SSL including the use of a trusted and valid certificates along with avoiding legacy versions of TLS/SSL with known vulnerabilities.

  3. Vulnerable services - The deployment of vulnerable services on the cloud endpoint includes the use of cleartext authentication, misconfigured services, exploitable services, or use of unsupported legacy operating systems as the host for the cloud endpoint.

Network Communication

The network communication properties are based on the observed network traffic between the three components, which are the smart-home device, the mobile application, and the cloud endpoint. There are three areas to inspect:

  1. Protocols - The use of third-party DNS, HTTP, UPnP, NTPv3, or custom protocols are considered under the protocol category. These protocols have security implications shown under the attack scenario section.

  2. Susceptibility to man-in-the-middle (MITM) attack - Identifies whether the communication between device-to-cloud, mobile application-to-cloud, or mobile application-to-device can be MITM attacked.

  3. Use of Encryption - Identifies whether the communication between device-to-cloud, mobile application-to-cloud, or mobile application-to-device uses or lacks encryption.

Threat Model

Attacker Types

The threat model assumes a network-based attacker ranked based on the following variations (high to low risk):

  1. Off-path attacker (Internet)

    1. The off-path attacker does not require direct access to the network where devices are deployed and can use n-day vulnerabilities or known flaws to mass exploit devices. These type of attackers are the most dangerous because of their capability of mass exploitation of vulnerable devices, mobile applications, cloud endpoints, and network protocols. (Internet)

  2. On-path attacker (Local Network)

    1. The on-path attacker is an attacker who has a presence on the network that the devices are deployed and can carry out direct attacks.

  3. Geographically proximity attacker (Next Door Neighbor)

    1. Geographically proximity attacker is an attacker whos physical presence is near to the deployed device and can carry out attacks against the initial device setup or using low-energy medium, such as Bluetooth, Zigbee, or ZWave.

Attack Examples

  1. Device

    1. Internet Pairing - A nearby attacker (type 3) hijacks the configuration setup over insecure Wifi or low-energy (Bluetooth, Zigbee, ZWave) protocols. A device requiring manual input of credentials or wired Internet connection is more secure.

    2. Configuration - An attacker (type 1 or 2) knows about weak device default configurations and uses this information to attack the device. A device that requires configuration before operating is more secure.

    3. Upgradability - An attacker (type 1, 2, or 3) can target vulnerable outdated devices that require manual or consent based upgrades. A device that automatically updates is more secure.

    4. Exposed Services - An attacker (type 1 or 2) has a bigger attack surface against a device with many running services. A device that uses a client model that runs no services is more secure.

    5. Vulnerabilities (CVSS) - An attacker can use one or many vulnerabilities in the device to expose sensitive information or gain control of the device. The vulnerabilities are in four categories (low, medium, high, and critical). The critical category means the device has an active vulnerability that has been exploited. The high category means the device has a serious vulnerability but has not been exploited yet. The medium category means that the device has misconfiguration issues that could lead to information disclosure or device compromise. The low category means the device has minor issues like running legacy protocols or debug reporting is turned on.

  2. Mobile Application

    1. Sensitive Data - An attacker (type 1, 2, or 3) can extract private APIs or secret keys to gain privilege on a device or a cloud endpoint. A mobile application that encrypts and stores its sensitive data is more secure.

    2. Programming Issues - An attacker (type 1, 2, or 3) can exploit incorrect initialization of a cryptographic protocol to disclose sensitive information. A mobile application that adheres to correct practices is more secure.

    3. Excess Permissions - An attacker (type 2) can utilize excess permissions to disclose sensitive data about the end-users. A mobile application that requests only needed permissions is more secure.

  3. Cloud Endpoints

    1. Domain Categories - An attacker has a larger attack surface as the number of endpoints increase. Additionally, the risk of exposing private information or having privacy implications is higher when vendors use third-party resources. A large number of first-party endpoints increases the attack surface and expose the device to higher risk. Hybrid cloud endpoints run the risk of exposing user information to cloud providers. Additionally, they can suffer from outages that the vendor cannot control. Third-party cloud endpoints increase the risk of privacy implication. The more parties involved the higher risk of privacy implication.

    2. TLS/SSL Issues - An attacker can exploit weaknesses in public-key infrastructure-based communication like TLS/SSL to snoop or compromise the integrity of the communication. Self-signed certificates can risk impersonation by an attacker, especially if the endpoints do not implement certificate pinning. Name mismatch on a certificate indicates the incorrect configuration of TLS/SSL services, which can be exploited by an attacker. Vulnerable versions of TLS/SSL can leak information about the encrypted content, which can be used by an attacker to snoop or modify the communication between two parties.

    3. Vulnerable Services - An attacker can exploit vulnerable services on the cloud to gain control over smart-home devices remotely or infer sensitive information. Old unsupported operating systems (OS) can suffer from vulnerabilities that developers no longer support, which leaves the cloud endpoint exposed to attackers. Information disclosure from misconfigured cloud services can leak sensitive information about the services that will help attackers in crafting an effective attack. Cleartext authentication can be snooped by attackers (type 1 and 2) on the network and used to gain unauthorized access. Exploitable services can be targeted by attackers to gain unauthorized access to a cloud endpoint and control smart-home devices.

  4. Network Communication

    1. Third-party DNS - A third-party DNS provider can infer usage patterns and cause privacy implications to end-users. Devices that use local DNS services can be configured securely and end-users gain more control.

    2. HTTP - An attacker can snoop and actively modify HTTP connections since they do not offer integrity or confidentiality. Components that use HTTPS are more secure.

    3. UPnP - An attacker (type 2) can issue commands and control devices that use UPnP because UPnP does not provide authentication. Devices that opt-out of UPnP and use alternative controls (via HTTPS) are more secure.

    4. NTPv3 - An attacker (type 2) can modify NTP version 3 or lower protocol responses, which can break certificate-based security. Devices that use NTPv4 are more secure.

    5. Custom - Custom protocols are non-standard and can be weak based on an overlooked flaw. Relying on security by obscurity is a bad practice. Devices that use community-vetted and standardized protocols are more secure.

    6. Man-in-The-Middle Attack - An attacker can intercept communication between smart-home device components and modify the content, which could result in a compromise. Components that verify endpoints and pin certificates are more secure.

    7. Encryption - An attacker can snoop on communication between smart-home components and infer sensitive information. Components that use encryption across all of their external communication are more secure.

Scoring Rubric

The scoring rubric outlines the weight distribution per property for each component. These can be reconfigured to emphasize important components across each deployment and their environment.

Device (42 Points)

The device component score is out of 42 points. The scoring system is inverted so that higher scores signify worst grade and lower scores signify better grade.

  1. Internet Pairing (3 points)

    1. Internet pairing refers to the device setup where local-network credentials are passed to the device to connect to the Internet. The 3 points represent the following from high risk to low risk:

      1. Wifi - device broadcasts unsecured wifi to allow users to connect and configure (3 points)

      2. Low-energy (LE) - device uses LE protocol to pair with mobile to device to configure (2 points)

      3. Wired - device uses a wired medium to directly connect to the local network (1 point)

      4. Manual - device requires users to manually input network credentials to connect and configure the device (0 points)

  2. Configuration (7 points)

    1. Configuration refers to the device setup phase where the device requires to be configured before operating or default configurations are acceptable.

      1. Default configuration (7 points)

      2. Forced configuration (0 points)

  3. Upgradability (4 points)

    1. Upgradability examins if the device requires manual, consent-based, or automatic updates. The 4 points represent the following from high to low risk:

      1. Manual (4 points)

      2. Consent (1 point)

      3. Automatic (0 points)

  4. Exposed services (4 points)

    1. Exposed services are services that run on the device and can be accessed directly from the local network. The 4 point bin distributions are the following from high to low risk:

      1. 5 or more services (4 points)

      2. 3-4 services (3 points)

      3. 1-2 services (2 points)

      4. No services (0 points)

  5. Vulnerabilities (24 points)

    1. Vulnerabilities are scored based on CVSS categories of low, medium, high, and critical. For each category, the scores are in the bins of 1-5, 6-10, and 11 or more. The point bin distribution is the following from high to low risk:

      1. Critical - 11 or more (10 points), 6-10 (9 points), 1-5 (8 points)

      2. High - 11 or more (7 points), 6-10 (6 points), 1-5 (5 points)

      3. Medium - 11 or more (4 points), 6-10 (3 points), 1-5 (2 points)

      4. Low - 11 or more (3 points), 6-10 (2 points), 1-5 (1 points)

Mobile (13 points)

The mobile component score is out of 13 points. The scoring system is inverted so that higher scores signify worst grade and lower scores signify better grade.

  1. Sensitive Data (6 points)

    1. Sensitive data are counted per piece of information and their bin distributions are the following from high to low risk:

      1. 6 or more (6 points)

      2. 3-5 (5 points)

      3. 1-2 (4 points)

  2. Programming Issues (4 points)

    1. Programming issues refer to any incorrect implementation or use of libraries.

  3. Over-privileged (3 points)

Cloud (92 points)

The cloud component score is out of 92 points. The scoring system is inverted so that higher scores signify worst grade and lower scores signify better grade.

  1. Domain Categorization (12 points)

    1. Domain categorization is divided into three categories and their bin distributions are the following ranked from high to low risk:

      1. Third-party domains - 76 or more (5 points), 26-75 (4 points), 1-25 (3 points)

      2. Hybrid domains - 46 or more (4 points), 16-45 (3 points), 1-15 (2 points)

      3. First-party - - 46 or more (3 points), 16-45 (2 points), 1-15 (1 points)

  2. TLS Configuration (30 points)

    1. TLS configuration scores the certificate, ciphers, and key-exchange algorithms used by the TLS service. The scores are in three categories and they are the following ranked from high to low risk:

      1. Self-signed certificate (10 points)

      2. Name-mismatch certificate (10 points)

      3. Vulnerable ciphers and KEA (10 points)

  3. Services (50 points)

    1. Services on the cloud endpoints are scored based on the weaknesses in the following 4 categories ranked from high to low risk:

      1. Exploitable service (14 points)

      2. Cleartext authentication (13 points)

      3. Information disclosure (12 points)

      4. Unsupported OS (11 points)

Network (28 points)

The network component score is out of 28 points. The scoring system is inverted so that higher scores signify worst grade and lower scores signify better grade.

  1. Protocols (8 points)

    1. Protocols are graded based on use in the following five categories ranked from high to low risk:

      1. Use of 3rd-party DNS (2 points)

      2. Use of non-standard custom protocol (2 points)

      3. Use of UPnP (2 points)

      4. Use of HTTP (1 point) ** HTTPS!=HTTP

      5. Use of NTPv3 (1 point) ** using version 3 or below.

  2. MITM Attack (10 points)

    1. MITM attack scores are based on the communication between three network directions, device-to-cloud, mobile app-to-cloud, and mobile app-to-device. The scores are given if the communication was successfully MITM’d during the evaluation. They are ranked from high to low risk:

      1. Device-to-Cloud (4 points)

      2. Mobile Application-to-Cloud (4 points)

      3. Mobile Application-to-Device (2 points)

  3. Network Encryption

    1. Encryption is scored on three categories and three score distribution. The three score distribution is no encryption, partial encryption, or full encryption and the three communication categories are the following ranked from high to low risk:

      1. Device-to-Cloud none (4 points), partial (2 points), full (0 points)

      2. Mobile Application-to-Cloud none (4 points), partial (2 points), full (0 points)

      3. Mobile Application-to-Device none (2 points), partial (1 points), full (0 points)

Score Calculation

The score calculation is an aggregate of the total points in each category divided by the total possible points. We use an inverse scoring system, where higher scores mean worst security posture. To generate the scores give to each device, we subtract the score fraction from one, which gives us the assigned scores. We use the general cutoffs to assign a grade letter (A - 0.9+, B - 0.8+, etc.). To calculate the score follow these steps:

  1. For each component (device, mobile, cloud, network), sum up the points.
  2. Divide the sum by the total number of possible points for the component category.
  3. Subtract the results from one.
  4. Apply grade assignment based on the letter cutoffs.

Example

Device A:

  1. In the device category it got 7 points
  2. In the mobile category it got 1 point
  3. In the cloud category it got 28 points
  4. In the network category it got 10 points

The scores are the following:

  1. Device score: 1-(7/42) = 0.83333
  2. Mobile score: 1-(1/13) = 0.92308
  3. Cloud score: 1-(28/92) = 0.69565
  4. Network score: 1-(10/28) = 0.64286

The grade assignments are the following:

  1. 0.83333 => 83.33% gets a B
  2. 0.92308 => 92.31% gets an A
  3. 0.69565 => 69.57% gets a D
  4. 0.64286 => 64.29% gets a D