Matrix performs well in BSI audit conducted by MGM Security Partners, only for findings to be misunderstood.
On Sunday 1 September, Heise published an inaccurate article that caused much confusion and – unfortunately – turned a positive story about the German government funding open source security research into a bad news story. This blog post aims to address the misunderstandings.
*Many thanks to Heise for subsequently correcting the headline
Germany’s Federal Office for Information Security (BSI) funds an initiative called Code Analysis of Open Source Software (CAOS). It’s a positive initiative that encourages the adoption of open source software by analysing it to strengthen confidence in the security properties of a particular project, and dispel any doubts about the correctness of the functional descriptions.
As part of CAOS activity, MGM Security Partners conducted a static code analysis of various implementations of Matrix, the open standard for secure and decentralised communication. MGM Security Partners conducted its research on the following versions and timeline:
- Synapse and Element releases on 2023-08-15
- Testing activities 2023-09-01 to 2023-11-17
- Scans ran during September and October 2023
This type of analysis uses tooling and manual testing to identify software flaws, vulnerabilities and bugs which then need to be evaluated for their impact and correctness. So we’re delighted that the BSI report did not identify any serious vulnerabilities in Matrix.
The main takeaways (full report here) were three low-severity security vulnerabilities and comments about ‘messy’ codebase evolution – which turned out to be based on code repetition and almost completely concerned test code, where such repetition is expected.
As you would imagine, the BSI’s CAOS process ensures that vulnerabilities are responsibly disclosed and both Matrix and Element welcomed the feedback. Element and Matrix shared their interpretation and resulting plans, and then acted upon the findings.
Unfortunately what should be an opportunity to congratulate the BSI and the German Government for generously investing in security audits of open source projects – and a moment to recognise that Matrix performed really well in the audit – turned into a jumbled misunderstanding that we’ve had to address publicly here.
A bit of unintentional CAOS…
The BSI press release issued on 30 August states:
“Kritische Schwachstellen hat das BSI den betroffenen Entwicklern sofort mitgeteilt. Diese haben die Schwachstellen analysiert und bereits reagiert.”
‘The BSI immediately notified the affected developers of critical vulnerabilities. They have analysed the vulnerabilities and have already reacted.’
There were no Matrix-related “Kritische Schwachstellen” (critical vulnerabilities) to report or mitigate – indeed we explicitly clarified this paragraph with BSI before it published the report – but unfortunately Heise misinterpreted the statement and ran an incorrect headline that read “BSI deckt schwerwiegende Sicherheitslücken in Matrix und Mastodon auf” (“BSI uncovers serious security vulnerabilities in Matrix and Mastodon”). The headline was simply wrong, with the report only finding three low-severity security vulnerabilities.
We also contacted BSI asking them to fix an error that incorrectly states:
“Außerdem wurden während der Codeanalyse in Absprache mit den Entwicklern kritische CVEs bearbeitet und publiziert.”
“In addition, critical CVEs were edited and published during the code analysis in consultation with the developers”)
As, again, there are no kritische (critical) CVEs.
Considerably less mess
As for the “messy” codebase, the paragraph “Gewachsene, unaufgeräumte Code-Basis” says:
“Sowohl bei Mastodon als auch bei Matrix konnten die Prüfer keine Hinweise auf ein strukturiertes, toolgestütztes Vorgehen zur regelmäßigen Identifikation und Korrektur von Schwachstellen finden. Die relativ große Menge an Code-Duplizierungen per Copy-and-Paste deuteten zudem auf teilweise chaotisch "gewachsene", unaufgeräumte Projekte hin.”
“At both Mastodon and Matrix, the auditors were unable to find any evidence of a structured, tool-supported procedure for the regular identification and correction of vulnerabilities. The relatively large amount of code duplication using copy-and-paste also indicated that some projects had ‘grown’ in a chaotic, untidy manner.”
Looking at the referenced “raw/synapse-cpd.txt” file around 90% (353 out of 393 instances) of the duplication found is in Synapse’s test suite – not Synapse itself. Code duplication in test suites is considered acceptable industry practice, given it doesn’t run in production, and comprehensive testing is inherently repetitive and duplicative.
In practice, we use a wide range of automated tooling to maintain the quality of our code, which anyone can see – e.g. GitHub’s Dependabot system to ensure dependencies are kept patched, and code quality tooling such as ruff and clippy.
Keeping secure
It’s worth noting that the code itself is only one aspect of maintaining a secure Matrix instance - you also need a well-maintained and supported distribution with which to deploy it; one that’s guaranteed to track best practices by the people who wrote the software in the first place.
Of the organisations who reached out to us yesterday to express concern about the (inaccurate) Heise article, many seemed to be improvising their own deployments. But large organisations – who already spend heavily on proprietary, and typically US, software – should also be investing in polished distributions of open source software. That is where scalable funding for open source comes from, and Element is only able to continue to maintain the security in Synapse and Element (and its contributions to the rest of Matrix) by heavy investment in our dedicated security, development and cryptography teams.
So: if you are a large organisation dependent on Matrix and are concerned about security, we would highly recommend using Element Server Suite as a fully supported commercial Matrix distribution, complete with security guarantees, and help fund our work on Matrix.
If an organisation is concerned enough to worry about the Heise article, they should be concerned enough to pay for the product they are using in a professional workplace.
Kudos to CAOS
As we said at the top of this blog post, CAOS is an excellent initiative and we’re delighted that the BSI is proactively auditing open source software. We’re very happy with how Element and Matrix performed in the audit, and look forward to more audits in the future.
Committed to open
Open source software developers are committed to transparency. So, for the record, below is an overview of all the report’s findings, starting with the three findings explicitly mentioned in the article and then listing out all the others:
E3.1
Report finding: Unauthenticated access to a file if you know the URL. Classified as ‘low’ severity.
Element response: Subsequently solved in Matrix 1.11 by authenticated media in June 2024.
E4.1
Report finding: Closing surveys belonging to other users
Element response: This is not a security vulnerability, as we fed back to BSI and the report authors. It’s a cosmetic bug which was misinterpreted as a vulnerability.
E6.1
Report finding: Bypass of CVE-2023-32683.
Element response: This is regarding a feature which is explicitly listed as not being a security mechanism in the Synapse documentation. We also believe it is a novel bug, not a bypass of CVE-2023-32683. MGM-SP has contacted us separately to inform us they are tracking it as CVE-2023-49953, but that CVE remains unpublished.
The other findings being:
C4
Report finding: CVE-2023-32683 not fixed correctly
Element response: same as finding E6.1.
D2
Report finding: Vulnerable dependencies
Element response: The vulnerable dependencies were either non-impactful or not in the application themselves but only in the accompanying toolchain. These toolchains only run on developers’ machines and never receive untrusted input, making the issues unexploitable.
E1.1
Report finding: Enumeration of usernames
Element response: This is a usability choice. There is rate limiting in place, as well as email verification and CAPTCHAs to prevent large-scale enumeration.
E2.1
Report finding: Weak password policy in Synapse
Element response: The researchers only analysed the client-side password policy in Element (a usability feature to avoid frustrating users making them try passwords until one fits the server-side policy). Synapse has its own password policy mechanism which is independent from the client, but for some reason this was not enabled during the test. See https://element-hq.github.io/synapse/latest/usage/configuration/config_documentation.html?highlight=minimum_length#password_config. The next-generation authentication component, Matrix Authentication Service (MAS), further improves on this.
E3.2
Report finding: Access to server version possible without authentication
Element response: We generally do not consider exposure of the Synapse server version as problematic since it is easy to identify the version via other means. The version is also used by tooling such as https://federationtester.matrix.org/ to detect federation configuration issues.
E5.1
Report finding: Logging of password reset information
Element response: The report states that a Matrix server administrator can use this information in order to reset a user’s password but the administrator is in a position to do so even without the information by virtue of being an administrator. Thus we don’t consider this a valid finding.
E7.1
Report finding: Long validity of the session
Element response: Classified as ‘low’ severity in the report. Fixed as part of our move to OIDC.
E8.1
Report finding: Upload allows any file format
Element response: That arbitrary files can be uploaded is intended, because the "media" repo is useful for general file transfers and attachments.
E9.1
Report finding: No virus scan when uploading files
Element response: When using the content scanner service, Element deliberately scans files at download, not upload, in order to apply current antivirus definitions rather than the ones in place at the point of upload
E10.1, E11.1 and E12.1
Report finding: No X-Frame-Options header, HSTS header not implemented, and MIME sniffing attacks potentially possible
Element response: Element Web is a static page with no server-side control. The server hosting it must set the headers according to the documented best practices.
F1
Report finding: Large number of TODO/FIXME comments in Synapse
Element response: TODO/FIXME comments are a typical practice and do not constitute vulnerabilities. Use of such comments depends heavily on a project’s workflow so some projects may have more than others, but this cannot be taken as a serious indicator of codebase quality.
F2
Report finding: Duplicates in code
Element response: 90% of the findings are about test code, which is expected to have duplication.
