386 lines
18 KiB
Plaintext
386 lines
18 KiB
Plaintext
From: "ICSE 2022 HotCRP" <noreply@icse2022.hotcrp.com>
|
||
Subject: [ICSE 2022] Reviews Ready: Authors' Response by Nov 13 - Paper #418
|
||
"Building a Secure Software Supply Chain..."
|
||
To: Ludovic Courtès <ludovic.courtes@inria.fr>
|
||
Date: Mon, 8 Nov 2021 20:33:17 +0000 (UTC) (3 days, 17 hours ago)
|
||
Reply-To: icse2022chairs@cispa.de
|
||
|
||
Dear Ludovic Courtès,
|
||
|
||
Thank you for your submission to the ICSE 2022 Technical Track
|
||
|
||
Building a Secure Software Supply Chain with GNU Guix
|
||
|
||
The author response period is now open with the deadline for response by
|
||
|
||
Saturday November 13, 2021 (11:59pm, AoE).
|
||
|
||
The reviews for your paper are attached to this mail. A pretty-printed
|
||
version can be retrieved at
|
||
|
||
https://icse2022.hotcrp.com/paper/418
|
||
|
||
|
||
What's new with ICSE 2022 Reviews and Authors' Responses
|
||
--------------------------------------------------------
|
||
|
||
* Reviews are organized along detailed *review criteria* (see #1, below)
|
||
* Reviewers have been instructed to *list only questions that they
|
||
think could alter their stance* (see #8, below)
|
||
* We have a *750 word soft limit* for authors' responses (see #10,
|
||
below).
|
||
|
||
|
||
About ICSE 2022 Authors' Responses
|
||
----------------------------------
|
||
|
||
1. Reviewers have been instructed to assess your paper by _five
|
||
criteria_, as detailed in the ICSE 2022 Reviewer Guidelines at
|
||
|
||
https://tinyurl.com/icse2022reviewing
|
||
|
||
notably Section 6 "Reviewing". Specifically, these criteria are:
|
||
|
||
* *Soundness*: The extent to which the paper's contributions and/or
|
||
innovations address its research questions and are supported by
|
||
rigorous application of appropriate research methods.
|
||
|
||
* *Significance*: The extent to which the paper's contributions can
|
||
impact the field of software engineering, and under which
|
||
assumptions (if any).
|
||
|
||
* *Novelty*: The extent to which the contributions are sufficiently
|
||
original with respect to the state-of-the-art.
|
||
|
||
* *Verifiability and Transparency*: The extent to which the paper
|
||
includes sufficient information to understand how an innovation
|
||
works; to understand how data was obtained, analyzed, and
|
||
interpreted; and how the paper supports independent verification or
|
||
replication of the paper's claimed contributions.
|
||
|
||
* *Presentation*: The extent to which the paper's quality of writing
|
||
meets the high standards of ICSE, including clear descriptions,
|
||
as well as adequate use of the English language, absence of major
|
||
ambiguity, clearly readable figures and tables, and adherence to
|
||
the formatting instructions.
|
||
|
||
2. Each review provides a score from "reject" (score: 1) to "accept"
|
||
(scores: 4 and 5). A "weak reject" (score: 2) and a "weak accept"
|
||
(score: 3) indicates that reviewers think they may still change their
|
||
score in light of other reviews, the authors' response, or the
|
||
discussion.
|
||
|
||
3. After the discussion following the author's response period, a
|
||
paper typically needs at least one champion to be considered for
|
||
acceptance - that is, at least one positive score (3 or higher).
|
||
|
||
4. Please study the reviews carefully and try to understand the
|
||
reviewers' concerns, as summarized in the fields on strengths and
|
||
weaknesses.
|
||
|
||
5. The authors' response is an opportunity for clarification. If you
|
||
believe the reviews contain factual errors, or that the reviewers
|
||
misunderstood your paper, please point this out, or include proposals
|
||
on how to improve your paper's presentation to address
|
||
misunderstandings as emerging from the reviews.
|
||
|
||
6. The review process remains double-anonymous. Please do *not*
|
||
provide additional links or clues that can disclose your identity, as
|
||
doing so could result in rejection of your paper.
|
||
|
||
7. Unless specifically asked for by reviewers, the authors' response
|
||
cannot include any new research results (e.g., additional experimental
|
||
results) that are not in the paper as submitted.
|
||
|
||
8. Reviewers provide “Questions for authors to respond to”, and it is
|
||
these questions your response should address. Reviewers have been
|
||
instructed to list **only questions that they think could alter their
|
||
stance**, and in decreasing order of importance. Hence, focus on
|
||
factual errors first, and then the top questions.
|
||
|
||
9. Authors' responses are optional, though providing a response will
|
||
enable the reviewers to further discuss your paper.
|
||
|
||
10. Reviewers need not read beyond the first 750 words of your
|
||
response; thus, be sure to start your response with the most important
|
||
answers and arguments. Words beyond the limit will be highlighted by
|
||
the HotCRP submission system.
|
||
|
||
11. Reviews may be updated after the end of the response period to
|
||
reflect the discussion among reviewers and take your response into
|
||
account.
|
||
|
||
|
||
How to Submit ICSE 2022 Authors' Responses
|
||
------------------------------------------
|
||
|
||
To submit your response, log into HotCRP:
|
||
|
||
https://icse2022.hotcrp.com/paper/418
|
||
|
||
and click on the "Add Author-Comments response" at the
|
||
bottom. Markdown styling and LaTeX math are supported; feel free to
|
||
make use of enumerations and headings to structure your response.
|
||
|
||
When your response is ready, check the "The response is ready for
|
||
review" box and submit; otherwise, it remains as a draft and will not
|
||
be visible to the reviewers.
|
||
|
||
The program committee members will read your response carefully and
|
||
take it into account during the discussions.
|
||
|
||
We thank you again for your submission to ICSE 2022!
|
||
|
||
Best regards,
|
||
|
||
Daniela Damian + Andreas Zeller
|
||
ICSE 2022 Program Co-Chairs
|
||
|
||
|
||
--
|
||
|
||
Review #418A
|
||
===========================================================================
|
||
|
||
Overall merit
|
||
-------------
|
||
3. Weak accept
|
||
|
||
Paper summary
|
||
-------------
|
||
# Building A Secure Software Supply Chain with GNU Guix
|
||
|
||
## Summary
|
||
Supply chain attacks are an ongoing concern in software engineering. Numerous attacks have
|
||
been successful so far and have resulted in leak of sensitive data, privacy, and financial
|
||
damage. Hence, preventing such supply chain attacks are important. This paper presents one
|
||
of the case studies in using a secure update frame with GNU Guix.
|
||
|
||
GNU Guix is a part of a new family of Linux distributions/deployment tools (along with
|
||
Nix) that aims at producing reproducible builds by removing nondeterministic parts of the
|
||
build process (also called functional package management). With such systems, the idea is
|
||
to capture all dependencies of a build artifact such that the artifact can be recreated bit
|
||
by bit given the same inputs. This means that Guix is source centric, which makes the
|
||
recommended update frameworks such as TUF (which focus on binaries) difficult to apply.
|
||
|
||
The solution found here was to rely on a chain of trust based on the git version control
|
||
system and signing the commits. The idea is to start with a trusted `.guix-authorizations`
|
||
that list the keys of developers who are allowed to commit to a repository. These developers
|
||
can then add or remove new collaborators at later signed commits, who are then given the same
|
||
privilege, and each commit can then be authenticated to have come from trusted sources.
|
||
|
||
This mechanism prevents both teleportation, and downgrade attacks by ensuring only
|
||
later commits are pulled into a system. The system, Guix, has deployed this framework for over
|
||
a year (since June 2020), and the downgrade protection has already proved useful.
|
||
|
||
Strengths
|
||
---------
|
||
* Provides a detailed report of secure update system as implemented in GNU Guix, which is sufficient to understand the properties of the system.
|
||
* The described system is sound, and has been deployed in the world.
|
||
|
||
Weaknesses
|
||
----------
|
||
* The presentation can be improved. In particular, the introduction is very limited and gives the reader no idea of the system being described.
|
||
* It would have been good to understand whether other source based systems such as BSDs and Nix have a secure update policy in place, and if so, in what respects they are lacking.
|
||
|
||
Comments for authors
|
||
--------------------
|
||
## Soundness
|
||
|
||
The described system for secure source updates is sound, with no obvious flaws. The system has
|
||
been in production for over a year with no vulnerabilities found, which improves the confidence
|
||
in its soundness. There are no research questions being evaluated per se, but as an experience
|
||
report, I believe that the techniques described in this paper are sound.
|
||
|
||
## Significance
|
||
|
||
Supply chain attacks are an ongoing concern, and verifiability of updates is an important problem.
|
||
Systems such as Nix and Guix are fundamentally source based distributions which are different
|
||
from binary distributions such as apt. Hence, solutions that apply to binary based distributions
|
||
do not necessarily apply to these systems. The systems such as Guix and Nix is important in their
|
||
approach to provide reproducible builds. Hence, the security of such systems is of concern in software
|
||
engineering. Hence, this contribution is significant in improving the state of the art in SE practice.
|
||
|
||
## Novelty
|
||
|
||
The described method has not been used in other projects, and hence, the process described is novel.
|
||
|
||
## Verifiability
|
||
|
||
Since this is an experience paper describing the update mechanism, there is little to verify.
|
||
Furthermore, Guix is open source, and the lack of vulnerabilities in its updates can be verified.
|
||
|
||
## Presentation
|
||
|
||
The paper is well written. However, the structuring is somewhat confusing. The introduction is just
|
||
two paragraphs (excluding listing of sections), and gives no indication of what the "secure update mechanism"
|
||
is.
|
||
|
||
Questions for authors to respond to
|
||
-----------------------------------
|
||
* Say there is a malicious commit being pushed from one of the authorized developers (the laptop was stolen, and the commit was pushed before the developer could revoke the key). This is then detected later. Is there a mechanism by which such a commit can be flagged? and the users who have already applied this update downgrade it securely and automatically? or does it require manual intervention?
|
||
|
||
|
||
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
|
||
|
||
|
||
Review #418B
|
||
===========================================================================
|
||
|
||
Overall merit
|
||
-------------
|
||
2. Weak reject
|
||
|
||
Paper summary
|
||
-------------
|
||
- Topic is relevant to ICSE
|
||
- It is a sound implementation of the authorization and authentication of commits
|
||
|
||
Strengths
|
||
---------
|
||
- Topic is relevant to ICSE
|
||
- It is a sound implementation of the authorization and authentication of commits
|
||
- Clear practical contribution of the paper
|
||
|
||
Weaknesses
|
||
----------
|
||
* Validation of the approach does not follow the scientific approach
|
||
* The extension is not related to the title and focus of the paper. *
|
||
The paper focus too much on the design and implementation of Guix
|
||
authentication and authorization mechanisms and does not focus on
|
||
the topic of the title. * The research questions of the paper are not
|
||
clear. * Research contribution is not clear
|
||
|
||
Comments for authors
|
||
--------------------
|
||
• Soundness: The paper is a practical implementation of an
|
||
authentication and authorization mechanism for Guix that can
|
||
provide effects to a secure software supply chain. But there
|
||
is a long way to get to build a secure software supply
|
||
chain. The developers’ authentication and authorization is
|
||
one of the weak links that needs to be addressed. The
|
||
implementation rational is well explained in the paper. But
|
||
the validation of the effectiveness of the approach is not
|
||
scientifically sound. The validation should show
|
||
scientifically the effects of the approach to the goals that
|
||
the authors have to the tool. As it is now, it is a nice
|
||
experience report of the implementation of the extension to
|
||
Guix and an anecdotal description of the possible
|
||
effects. The reproducibility of the study is not possible
|
||
with the way it is described. The scientific methodology for
|
||
the empirical study is not clear. • Significance: Software
|
||
developers needs proper and transparent ways to securely
|
||
develop software. Then the contribution is relevant to the
|
||
field, specially to practice. But it is not clear how the
|
||
contribution is relevant to the state of the art. •
|
||
Novelty: Adding authorization and authentication to the
|
||
commits is not per si a new concept, but the way the authors
|
||
have developed can potentially produce the effects claimed by
|
||
the authors. But the paper is not adding to the evidence based
|
||
knowledge on that because of the lack of rigour on the
|
||
validation of the implementation. • Verifiability and
|
||
Transparency: The study is not reproducible. The implementation
|
||
of the extension is well explained that it is possible to
|
||
reproduce if other researchers would like to do in another
|
||
tool. • Presentation: The title does not match the paper. The
|
||
authors are overstating on the title to the work that was
|
||
done. The validation needs to be scientifically performed.
|
||
|
||
|
||
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
|
||
|
||
|
||
Review #418C
|
||
===========================================================================
|
||
|
||
Overall merit
|
||
-------------
|
||
1. Reject
|
||
|
||
Paper summary
|
||
-------------
|
||
GNU Guix is a set of software deployment tools, and it includes a package manager. Guix is a “source-based” deployment tool, which means that every piece of software is built from source. The authors of the paper present a Git checkout authentication mechanism that can help preventing attacks, such as downgrade attacks and teleport attacks.
|
||
|
||
Strengths
|
||
---------
|
||
- The authors have deployed their approach for more than a year to experiment.
|
||
- Supply chain security is an important and timely topic.
|
||
|
||
Weaknesses
|
||
----------
|
||
- The contribution of the paper is not clear.
|
||
_ The paper needs to be restructured.
|
||
- The experiment result is incomplete.
|
||
|
||
Comments for authors
|
||
--------------------
|
||
Soundness: The soundness is low - no clear research questions or rigorous application of appropriate research methods.
|
||
Significance: The contributions are not clear. The paper does not
|
||
make clear the specific contributions of Guix over other solutions.
|
||
Novelty: The contributions are not clear, though I suspect there are
|
||
novel aspects to thew or. Verifiability and Transparency: The data
|
||
and code is not published.
|
||
Presentation: The paper needs to be restructured. The motivation for
|
||
needing a new tool needs to be clear at the start of the paper -- and
|
||
be substantiated throughout the paper. The threat models which are
|
||
spread throughout the paper should be consolidated.
|
||
Detailed comments
|
||
|
||
Abstract:
|
||
- The abstract is incomplete. The goal of the paper is not clear (focuses on the security of updates with Guix for what?). No mentioning of research that was done -- research approach or findings. Currently, it seems like an engineering, not a research paper.
|
||
|
||
Introduction:
|
||
- What is the motivation of your work in terms of shortcomings to other tools?
|
||
- What is the goal statement of the paper?, What are research questions that you want to answer?, What is the summary of your approach?, What are the contributions of the paper.
|
||
|
||
|
||
Background:
|
||
- Readers often skip background sections. I would not put the essential information about Guix in the background section (first two paragraphs).
|
||
|
||
Rationale:
|
||
- A research question is mentioned in the second paragraph. It is better to move it to the introduction.
|
||
- In the last sentence you said "These are the kind of attacks we want to protect against." Are these just examples of attacks or all the attacks? You mentioned three types of attacks but just the two of them have names, what about the first one?
|
||
- You might want to call this section "Threat Model".
|
||
|
||
Authenticating Git Checkouts:
|
||
- The section is not structured well. Is this a methodology section? It is not clear based on the title and the first paragraph of it.
|
||
- In the first paragraph, you mentioned that the problem is not
|
||
specific to Guix. In that case, Guix is just a case study of your
|
||
work. It is not clear based on your title, abstract, and the
|
||
research question. It seems now that authenticating git checkouts
|
||
may be the primary motivation for creating Guix since you say that
|
||
"no existing tools or protocols support[ing] off-line checkout
|
||
authentication". If that is the primary shortcoming of existing
|
||
tools, you should put that in the Introduction. - Are rules listed
|
||
in lines 257-261 your contributions? If yes, why in line 319 is it
|
||
mentioned that "This is what Guix pull implements"? If not, why in
|
||
line 255 it is mentioned "To implement that, we came up with the
|
||
following mechanism and rule"? It is not clear what is your
|
||
contribution and what is existing implementation?
|
||
|
||
Establishing trust:
|
||
- In line 441, why does A lack .guix-authorizations?
|
||
|
||
Downgrade attacks:
|
||
- What about other types of attacks you mentioned in lines 206-214? The one that has no name and teleport attack? Why is there no separate sections for them?
|
||
- Is this section your contribution or just what exists in the literature? It is not clear.
|
||
|
||
Mirrors and The Risk of Staleness:
|
||
- Same as the previous section, it is not clear which part is your contribution?
|
||
|
||
Implementation:
|
||
- You have missed the methodology section before this section. We still don't know your contribution, and what process you perform.
|
||
- The writing is lines 533-543 is not scientific. You can not make a choice because you "feel" something. You need justifications. This comment applies to some other parts of the paper as well.
|
||
- Unfortunately, results in Section 8.5 are incomplete. For example, Why you say "Downgrade prevention has had a more visible impact"? What was the frequency of these attacks before and after your implementation? You need more analysis/collected data to report here.
|
||
|
||
Related Works:
|
||
- What are the similarities and differences of any of these related works to your study?
|
||
|
||
Conclusion:
|
||
- Again, what is the justification to say "our experience so far has been positive"? You need reported data to substantiate that.
|
||
|
||
Questions for authors to respond to
|
||
-----------------------------------
|
||
1. I understand software supply chain is a major issue. But, what was missing from existing tools that motivated the creation of Guix? Please make that clear in the paper.
|
||
|