20 comments

  • lz40016 minutes ago
    A lot of this university research on AI feels dated. This is something we all faced months ago. Many times they're also reaching conclusions using obsolete models, etc. Feels like things move too fast for traditional research cycles to catch up to AI dev
    • thraway38379 minutes ago
      Exactly. What a weird clickbait title to come out of a Tier 1 institution. You have Harvard PhD&#x27;s concluding this!? This is what a PhD at that insitution is putting out? That is really sad. FTA:<p>The paper’s interviews show what that looks like on a team: a junior software engineer described a project where only one senior engineer could approve changes and could review the interviewee’s code only once a week.<p>So the issue is not AI. The issue is organizations and certain personalities who think that a human is going to be a better coder than an engineer. At this point rolling and writing your own code is akin to making up your own encryption scheme.
  • jacobgorm28 minutes ago
    The problem is that when nobody writes code or feels ownership over it anymore, nobody will be qualified to or really interested in doing the code reviews. The way forward could involve dropping code reviews altogether. You could argue that the useful pingpong between coder and reviewers has been replaced by the similar pingpong between the prompter and the agent.
    • prmoustache19 minutes ago
      In my company, people found out that if they use an agent that commits and do the pull&#x2F;merge request, they can approve it themselves and bypass a second set of eyes.<p>Since they are the one doing the prompting, they quickly got lazy of actually reviewing the output from the LLM which means that nobody is reviewing anything.<p>It hasn&#x27;t happened yet but I guess that is how you end up with major outages.
    • bambu15 minutes ago
      That would be great, IF there were valuable ping-pong between the prompter and the agent. For skilled engineers the game between themselves and the agent is enough. But most people are submitting senseless slop grenades for review.<p>Claude always comes up with more things to add to the code that are pure bloat and become inbred bombs over several iterations, and some people can&#x27;t distinguish what suggestions to reject and go along with everything Claude suggests to the detriment of the program #garrytangate
  • JonoW9 minutes ago
    Maybe we should just review the prompt(s) the PR author used. Would be nice if this was automated and included in a PR. So code review is really checking that the inputs were valid, and testing the outputs.
    • sdevonoes5 minutes ago
      For a given feature my list of prompts looks like:<p>- very well defined initial description of problem plus asking exactly what i want<p>- fix X<p>- why do you claim that Y?<p>- fix Y<p>- …<p>I don’t think it would be useful to review or commit prompts
  • dangoodmanUT31 minutes ago
    This feels obvious to me, but it&#x27;s interesting how so many people feel the opposite.<p>I guess it depends a lot on how a company manages their SDLC&#x2F;bureaucracy
    • michaelt20 minutes ago
      If someone doesn&#x27;t think code review is the bottleneck, chances are they&#x27;ve got AI doing their code reviews too.
    • fgblanch29 minutes ago
      Harvard researchers found a tree in a forest
  • gzell1 hour ago
    From real-world experience, I have to disagree. It always has been and continues to be a faulty, up-front specification problem. The tail don&#x27;t wag the dog.
    • wilkystyle3 minutes ago
      With all respect to your personal experience, I could not possibly disagree more strongly with this.<p>In my own real world experience, the bottleneck has never been crafting the perfect specification up front. In fact, I have never, not even once, seen the initial specification for any real world software project of any mentionable size or scale survive unscathed to delivery. In reality, it is much like the old wartime maxim. &quot;No Plan Survives First Contact With the Enemy&quot;.<p>The real world is messy and unpredictable. Every single software project of significance that I have been a part of has always involved testing, validating, and reworking the assumptions in the initial design and spec, refining them to arrive at the final product.<p>Furthermore, this iterative process (pre-AI) used to involve one or more humans following the development story and understanding what design changes and corresponding trade-offs were happening along the way (and why they were being made). By the time there was a finished deliverable ready for review, the person involved was privy to the story of why the development shaped up the way it did. They were able to account for the big picture, adapt accordingly along the way, and be able to explain why things ended up the way they did.<p>IME, with fully-agentic PRs, AI tends to view the specification as gospel and to view every point of friction between the specification and the real world as something to be worked around or minimized. Where a real human would pause, zoom out, and consider the system and bigger picture, considering what design changes might be needed, the AI will assume that its job is to make the specification a reality at all costs.<p>You can of course mitigate this by staying more closely involved with the work as it&#x27;s being done, but this is of course slower than just throwing a spec over the wall and letting an agent run with it. It seems that people are far too eager to optimize for raw speed, and not nearly concerned enough with correctness or human understanding.<p>A sister comment mentioned a plethora of PRs that don&#x27;t solve the right problem. If you are swamped with PRs solving the wrong problem, the root cause is not that you didn&#x27;t foresee every future possibility or undiscovered devil in the details in order to create the right spec, the problem is that you thought you could come up with the right prompt at the outset and that an agent living in a clean digital world could make the journey all the way to a product fit for the messy real world on its own.
    • donkey_brains36 minutes ago
      Very true. Which makes the conclusion reached be this study even worse: we are stuck reviewing endless PRs that solve <i>the wrong problems</i>.
    • diego_moita14 minutes ago
      I don&#x27;t fully disagree with you. That is a real big problem.<p>But both things can be true. One problem doesn&#x27;t exclude the other.
    • mathisfun12326 minutes ago
      &gt; The tail don&#x27;t wag the dog<p>If I increased the output of my pump jack and identified the refinery as the new bottleneck would you say the same pithy thing? <i>NB</i> a single pump jack can pump different grades of crude oil at the same time.
      • skydhash15 minutes ago
        You may need to check the pipeline so that the refinery isn&#x27;t stuck filtering random stuff from the oil.<p>AKA, it would be nice to have nice patches like this one to review.<p><a href="https:&#x2F;&#x2F;cgit.freebsd.org&#x2F;src&#x2F;commit&#x2F;?id=09d62ba33de113d01e889f5f8a5a8db071fba91e" rel="nofollow">https:&#x2F;&#x2F;cgit.freebsd.org&#x2F;src&#x2F;commit&#x2F;?id=09d62ba33de113d01e88...</a>
    • webdood904 minutes ago
      So the solution is: write a perfect spec and then you&#x27;ll have perfect code, no need for reviews?<p>What world do y&#x27;all live in where this is your reality?
  • nout7 minutes ago
    That&#x27;s sort of old news. Code isn&#x27;t manually reviewed anymore much.
  • wewewedxfgdf27 minutes ago
    Test don&#x27;t review.
    • vecter5 minutes ago
      What about design decisions or code quality? Tests don&#x27;t cover those.
      • wewewedxfgdf2 minutes ago
        I built a system that records numbered design decisions and numbered requirements as claude works in a Postgres database organised as a graph with links&#x2F;interconnection between items.<p>It&#x27;s an MCP server that Claude uses as we develop so there&#x27;s a permanent record of everything and work is driven by requirements and decisions.
    • webdood903 minutes ago
      Equally time consuming and mind-numbing, IMO.
    • avereveard21 minutes ago
      That bottlenecks too, in startup ahead of the adoption curve the whole cicd pipeline down to how to merge is being rethought
  • u1hcw9nx2 hours ago
    This again increases the demand for good programmers. Write only programmers are dime on dozen. Those who can read code written by others (and themselves) and analyze it are harder to find.
  • dominotw13 minutes ago
    No pull request in my company get any human review comments anymore. No one is reading.
    • Terr_11 minutes ago
      It&#x27;s also hard to blame reviewers, when often that phase is preceded by the supposed &quot;authors&quot; not truly reading the reviews.
  • judahmeek2 hours ago
    I&#x27;ve submitted the original paper (<a href="https:&#x2F;&#x2F;fion.ac&#x2F;jellyfish.pdf" rel="nofollow">https:&#x2F;&#x2F;fion.ac&#x2F;jellyfish.pdf</a>), but others already have to little engagement, likely due to the paper&#x27;s terrible title.
  • thom3 minutes ago
    I really think ‘shifted the bottleneck to code review’ is misleading. It’s bad code. They write bad code. Yes if you want to salvage it you do a lot of code review, but that’s only because you’ve set your process up to pick spaghetti off the wall against which it has been flung. You could easily just push it all live without review and then pretend the bottleneck is SRE time, or set fire to yourself and claim the bottleneck is emergency service response time. But the root cause is that it’s bad code.
  • ttctciyf24 minutes ago
    Vaguely reminiscent of the classic trope around 70&#x27;s computer &quot;effficiency gains&quot; where a newly introduced computerised accounting system would end up taking longer than the original human-run system because once the computer was through doing the books the original human bookkeeper must still check the results manually.
  • TexanFeller7 minutes ago
    &quot;code is cheap&quot;...if you don&#x27;t need to understand, review, and maintain it as part of a team.
  • ontouchstart23 minutes ago
    I have never heard of the term Job Market Paper (JMP) but this is the second one I saw on HN in two days. The other one is this<p><a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=50516245">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=50516245</a><p>Guess I have to ask AI.
  • jsrozner37 minutes ago
    We needed a research paper to tell us this?
    • david-gpu28 minutes ago
      Just because it is obviously true to you, it doesn&#x27;t mean it is obviously true to others.<p>And just because it is obviously true to you, it doesn&#x27;t mean that it is true, either.
      • aleqs23 minutes ago
        And just because harvard researchers say something doesn&#x27;t make it true either. Especially when it&#x27;s like 6 months late and reviews are probably not the bottleneck anymore - testing, ci, parallelization are (imo).
    • hazbot35 minutes ago
      A <i>Havard</i> research I&#x27;ll have you know! &#x2F;s
    • aleqs30 minutes ago
      do you even have a PhD?<p>(&#x2F;s)
  • root_axis33 minutes ago
    I&#x27;m being told the answer is simply to let AI review and merge the code. Problem solved.
  • avereveard22 minutes ago
    Harward researcher read lovable blog post
  • wewewedxfgdf28 minutes ago
    But people keep saying &quot;coding was never the bottleneck!&quot;.
  • inshard7 minutes ago
    In other news, Dartmouth researchers find the sky is blue.
  • furytrader20 minutes ago
    [flagged]