HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

> 35 points for open source contributions

> 30 for personal projects

I don't contribute to open source or have personal projects because I don't spend my free time doing what I do 40 hours a week to make a living. My 15 years of work experience is worth a maximum of 25%, so any company using this idiotic system would pass on me immediately. Open source and personal projects are fine, but in no sane world are they worth 65% of a resume's score.

Now all my "non-work" time is spent on startup work. And none of that is visible via GitHub.

Free software work doesn't imply we work for free. We work on our projects, the stuff that we actually enjoy working on. Nobody is going to work on corporate products without adequate compensation.

I guess there sadly are many nobodies who do this to hope to become somebody.

I wonder if that assumption is bourne out in reality though?

I'd imagine if someone's OSS contributions are enough of a factor that it's worth hiring them, they're not going to drop it on a whim to work extra hours on the day job.

(Assuming you weed out open source contributions like "I made a todo list app in React but licenced it as MIT" or "I fixed a typo in the docs for NextJS". )

As someone who’s run hiring pipelines for technical roles in the past few years, that’s actually a fantastic number. I objectively hate saying that, but it’s true.

35% chance of elevating a technical individual to the next stage with no effort? I’ve seen as many as 100+ applicants an hour even when including a domain specific screener question. That’s 35 “screened” applicants in an hour. Were valid candidates screened out? Yes. Does you still have a candidate pool 35x larger than you need? Unfortunately, also yes.

The volume of applicants is SO HIGH such that your chances of getting moved to the next stage are actually markedly worse if AI isn’t involved. If you didn’t apply immediately (using an AI bot) there’s 50+ people ahead of you, and an exhausted technical leader if they ever make it to your resume.

Referral bonuses exist for a reason.

Gates that reduce resume flow-through are only useful if their reduction is correlated with quality. Otherwise they're just dragging out your hiring process or unnecessarily causing you to ultimately lower your hiring bars.

The volume is infeasible to review everyone for quality, even at an hour scale. The conclusion and solution is inevitable, though I wish it were different. 35% is actually really good if you’re not coming in through a referral.

The current reality is <1% and the person reviewing you is exhausted.

Corpo bullshittery at its finest.

If you have 1000 applications for every job, and you know that a bunch of these applications are "a bad fit", to put it mildly, you have to filter. And you cannot realistically give every resume a good, human look. By the time HR would be done, the market has already moved on five times.

So, what is the real difference between being overlooked because HR could only look at the first 100 resumes, or the AI filtered all 1000 resumes down to 100? In the end, a fuckton of potentially great people get their feelings hurt either way.

Here's a realistic proposition. HR just wants to inflate numbers so that they seem busy looking for the right fit. Keep posting open for 1 week, manually filter for another week, invite people, employ one. Plenty of people with degrees looking for jobs right now, I don't see what's the issue with just trying one. Companies desperately look for the "magic" applicant that checks all boxes, while also trying to pay them almost minimum wage.

Instead of spending all those resources on resume filtering, hire resume blind. Instead of using llms for a thing they are bad at (subjective decision making) use them to build a deterministic process that isn’t.

Use work sample hiring as the filter. Make the work sample automatic to sign up for and judge.

At 10 seconds per resume, it would take you 3 hours to go through all 1000 resumes. I don't know what you consider "good" and "human", but my human eyes could easily do good enough, fully manual pre-screening at a rate of 1 requisition per day.

At 10 seconds per resume, I would not assume that you're screening better than the LLM.

What I hear happens now: people apply for a "Senior Golang SWE" with 2 years of experience with C#. Or, relevant for hiring in the US, the job posting says the visa is required, but people apply without it anyway.

Maybe a platform could be designed where candidates have one account for multiple companies, and the number of applications on the platform is limited to, say, ten per person per month or something. To get people to be selective. I don’t think this should be the only way to apply, but maybe the companies involved could look there first.

It’s all probabilities in the end. And if an LLM gives you more a more relevant pool vs random distribution, that’s still a net benefit.

*According to our proprietary, undisclosed, non-deterministic metric, which may or may not be Math.random

https://stackoverflow.com/questions/16833100/why-does-the-mo...

    Whenever I send them out
    The filters always route: 
    "Spammer: John Jacob Jingleheimer Schmidt"
    [N/A] [N/A] [N/A] [N/A]

1. Give them some easy leetcode questions. Nothing that a competent programmer would have any problem with.

2. If they pass, ask for a deposit of like $20. Shouldn't be an issue for people who are actually serious.

3. Do more simple leetcode questions but this time on zoom so you can tell if they are using AI. If they pass that they get the deposit back.

(Yeah I know there are real-time interview cheat AI programs but based on what I've seen on demos of them it's super obvious when they're being used.)

Probably not practical but just a thought!

If the first 50 people who apply are all bots, why are you reading resumes in order of submission?

> temperature 0.1 — low, supposedly nudging the model toward deterministic outputs

This is not correct (and is briefly touched on later in the piece when he sets temperature to 0), temperature is not some kind of "deterministic" switch, but rather it affects the sampling distribution (which becomes more "spiky"—but is still very much a distribution).

Show someone a list of resumes with an "applicant score*" and they'll naturally ignore the ones with a low ranking

*scores are generated with AI, mistakes may be made, use only as a guide and verify results

I don't know for sure, but I would be surprised if it was illegal in my particular US state. You might be able to argue the AI has inherent biases that introduce illegal discrimination in the hiring process, but my understanding is winning I case like that would be very difficult, especially since most employers are very cagey about their hiring process and why they mades a decision.

nonetheless, people will defend history as perfect and say those samples, like nepo babies, are "perfect".

It is a common misconception, but it is not true even in principle. If I have 2 or more logits which are equal to the maximum of my logits, I will sample uniformly random from them with any temperature, even zero. Sampling from softmax([1, 0, 1]) is still stochastic at temperature 0, because the limit is to sample uniformly from the first or the last element.

Anyway: "GPUs don't do deterministic matrix multiplications" is the biggest source of randomness in LLMs. GPUs put the associativity of the sums in matrix multiplications in arbitrary order, and this has a huge impact on the logits coming out of the neural network.

That’s user-controlled too, not an inherent property of GPUs:

https://docs.pytorch.org/docs/2.12/generated/torch.use_deter...

> torch.bmm() when called on sparse-dense CUDA tensors

And it's not listed under the operations that raise an exception otherwise, so I'm not sure the docs promise that dense-dense matrix-matrix products are deterministic.

But this isn't a fundamental property of LLMs, it's just an implementation detail. It's pretty obvious that if you evaluate the matrix multiplications correctly and deterministically sample from the highest-probability outputs, you will have a deterministic LLM.

Well, in theory theory, temperature 0 doesn't really exist. Mathematically, as lim temperature->0, the distribution gets spikier and spikier, the most likely sample goes to almost-but-not-quite infinity and the rest go to almost-but-not-quite 0. In practice, temperature=0 is literally a separate branch of an if statement that just picks the most common sample (using the actual formula that works for non-zero values would cause a zero division).

However, due to things such as batching and even different kinds of floating point imprecisions for different algorithm implementations, the probability distribution itself often differs run-by-run, so what you sample from it also differs.

It does exist very much, even if you go to pure math. Look at the softmax function and take the limit as T->0. It becomes a dirac-delta function. I.e. in a discrete setting (like for LLMs with a finite set of output tokens), probability P becomes one for argmax and 0 for everything else. Only in coding practice it is easer to implement T=0 as a simple if check that directly chooses argmax instead of calculating the limit of some function that includes 1/T quotients. But setting T to zero is in both, theory and practice, turning the usual probability function into greedy sampling.

In pure math, it does not always do that. It becomes a dirac-delta comb with equal weight on every maximum. There can be more than 1 maximum. Setting the temperature to zero turns into greedy sampling, but greedy sampling is not necessarily deterministic as you can have multiple equally optimal options.

Any two tokens ending up with the exact same logit is very unlikely, but not impossible; and as the number of output tokens grows, the odds that it will happen eventually gets higher and higher.

I suppose, to ensure determinism, rank by logit then token ID, so you still have a deterministic winner even if occasionally two tokens get precisely identical logits.

This is exceedingly unlikely, as training will only push one of them up for any individual sample. There are likely some pathological situations that could end up with that situation, maybe, but it is pretty unlikely in a general case.

If there's one counterexample, it's not really deterministic.

My point is, deterministic logic matters in certain circumstances 100% of the time. Forcing the LLM to make something unlikely is not good enough because a series of mistakes could very quickly bankrupt the company.

If your argument is that the danger of equal values being selected inconsistently breaks determinism, that's a trivial problem to solve.

Any non-infinite precision numbering system by definition is at the limits of it's precision when equal values occur. If you need to order such values you can extend the precision and add on a deterministically unique tiny value (position, order encountered, etc.) . Your original value stays in the same precision range but they are now unique.

It's usually more likely that you want to sacrifice a little precision for determinism so you can quantise to allocate the range where you apply the unique ID

For example if you had an array of 256 fp32 values but you required them to be unique, you can lop off 8 bits of mantissa and replace it with its index in the array, Every value is then unique.

Granted token dictionaries make for some fairly hefty indexes now, but the principle applies in general, it's easily solvable if you are prepared to spend some precision or do some extra calculation.