2. Heuristics for Completing Quantum Mechanics
The goal of any hidden variable theory [
3,
18,
19] is to reproduce the statistical predictions encoded in the quantum states
of physical systems using hypothetical dispersion-free states
that have no inherent statistical character, where the Hilbert space
is extended by the space
of hidden variables
, which are hypothesized to “complete” the states of the physical systems as envisaged by Einstein [
2]. If the values of
can be specified in advance, then the results of any measurements on a given physical system are uniquely determined.
To appreciate this, recall that expectation value of the square of any self-adjoint operator
in a normalized quantum mechanical state
and the square of the expectation value of
will not be equal to each other in general:
This gives rise to inherent statistical uncertainty in the value of
, indicating that the state
is not dispersion-free:
By contrast, in a normalized dispersion-free state
of hidden variable theories formalized by von Neumann [
3], the expectation value of
,
by hypothesis, is equal to one of its eigenvalues
, determined by the hidden variables
,
so that a measurement of
in the state
would yield the result
with certainty. How this can be accomplished in a dynamical theory of measurement process remains an open question [
18]. But accepting the hypothesis (
3) implies
Consequently, unlike in a quantum sate
, in a dispersion-free state
observables
have no inherent uncertainty:
The expectation value of
in the quantum state
can then be recovered by integrating over the hidden variables
:
where
denotes the normalized probability distribution over the space
of thus hypothesized hidden variables. The quantum mechanical dispersion (
2) in the measured value of the observer
can thus be interpreted as due to the distribution
in the values of the hidden variables
over the statistical ensemble of the physical systems measured. Moreover, the Born rule can also be recovered using this prescription. If the system is in a quantum state
and the observable
satisfies the eigenvalue equation
, then, using (
6) and the eigenvalues
and 0 of the corresponding projection operator
, the probability of observing the eigenvalue
of
can be recovered as
The probabilities predicted by the Born rule can thus be interpreted as arising from the statistical distribution of
.
As it stands, prescription (
6) amounts to assignment of unique eigenvalues
to
all observables
simultaneously, regardless of whether they are actually measured. In other words, according to (
6) every physical quantity of a given system represented by
would possess a unique preexisting value, irrespective of any measurements being performed. The prescription (
6) thus mathematically encodes Einstein’s conception of realism. In [
2], Einstein explained his point of view in terms of the position and momentum of a free particle as follows: “The (free) particle really has a definite position and a definite momentum, even if they cannot both be ascertained by measurement in the same individual case [as in quantum mechanics]. According to this point of view, the
-function represents an incomplete description of the real state of affairs.” The acceptance of this point of view — Einstein continues — “would lead to an attempt to obtain a complete description of the real state of affairs
as well as the incomplete one (my emphasis), and to discover physical laws for such a description.” Accordingly, the left-hand side of the first equality in (
6) provides the incomplete description of the system and its right-hand side provides the complete one, with all possible statements one can make about the system encoded in the expectation values of the observables being measured in the state of the system [
3]. If
and
are two non-commuting observables, then the uncertainty relation between them can also be interpreted as
in terms of the probability distribution
in the values of the hidden variables
, where the first inequality is the one established by Robertson [
20], and
is a purely imaginary eigenvalue of the skew-Hermitian operator
. Similarly, using the kinematical equivalence seen in (
6), the dynamical equivalence between the quantum mechanical description and Einstein’s “complete” description can also be established, as demonstrated in the
Appendix A below:
where
H is a Hamiltonian operator in
and
is a classical Hamiltonian function in the corresponding phase space. If particular values of
are precisely known with
, then the right-hand side of (
9) would reduce to the classical Hamiltonian equations of motion. Otherwise, Ehrenfest’s equation in quantum mechanics on the left-hand side of (
9) can be understood as an ensemble average of classical dynamics over the probability distribution
of
. Thus, once the hypothesis (
3) regarding the dispersion-free states
is accepted, each probabilistic statement about the quantum system, (
6)–(
9), can be traced back to the incompleteness of our knowledge about the system.
In Section 2 of [
18], Bell works out an instructive example to illustrate how the prescription (
6) works for a system of two-dimensional Hilbert space. It fails, however, for Hilbert spaces of dimensions greater than two, because in higher dimensions degeneracies prevent simultaneous assignments of unique eigenvalues to all observables in dispersion-free states
dictated by the ansatz (
3), giving contradictory values for the same physical quantities. This was proved independently by Bell [
18], Kochen and Specker [
21], and Belinfante [
22], as a corollary to Gleason’s theorem [
23,
24].
These proofs—known as the Kochen-Specker theorem—do not exclude contextual hidden variable theories in which the complete state
of a system assigns unique values to physical quantities only
relative to experimental contexts [
19,
24]. If we denote the observables as
with
c being the environmental contexts of their measurements, then thenon-contextual prescription (
6) can be easily modified to accommodate contextual hidden variable theories as follows:
Each observable
is still assigned a unique eigenvalue
, but now determined cooperatively by the complete state
of the system and the state
c of its environmental contexts. Consequently, even though some of its features are no longer intrinsic to the system, contextual hidden variable theories do not have the inherent statistical character of quantum mechanics, because outcome of an experiment is a cooperative effect just as it is in classical physics [
24].Therefore, such theories interpret quantum entanglement at the level of the complete state
only epistemically.
For our purposes here, it is also important to recall that in the Hilbert space formulation of quantum mechanics [
3] the correspondence between observables and Hermitian operators is one-to-one. Moreover, a sum
of several observables such as
is also an observable representing a physical quantity, and consequently the sum of the expectation values of
is the expectation value of the summed operator
,
regardless of whether the observables are simultaneously measurable or mutually commutative [
18]. The question then is, since within any contextual hidden variable theory characterized by (
10) all of the observables
and their sum
are assigned unique eigenvalues
and
, respectively, would these eigenvalues satisfy the equality
in dispersion-free states
of physical systems in analogy with the linear quantum mechanical relation (
11) above? The answer is: Not in general, because the eigenvalue
of the summed operator
is not equal to the sum
of eigenvalues
for given
, unless the constituent observables
are mutually commutative. In other words,
in general, and therefore the correct counterpart of relation (
11) is not (
12) but
As Bell points out in Section 3 of [
18], the linear relation (
11) is an unusual property of quantum mechanical states
. There is no reason to demand it
individually of the dispersion-free states
, whose function is to reproduce themeasurable features of quantum systems only when averaged over, as in (
10). There is no reason why the value of
should not be determined by some
nonlinear function
. I will come back to this issue in
Section 5.
In [
18], Bell explains this non-linearity using spin components of a spin-
particle. Suppose we make a measurement of the component
of the spin with a Stern-Gerlach magnet suitably oriented in
. That would yield an eigenvalue
of
as a result. However, if we wish to measure the component
of the spin, then that would require a different orientation of the magnet in
, and would give a different eigenvalue,
of
, as a result. Moreover, a measurement of the sum of the
x- and
y-components of the spin,
, would again require a very different orientation of the magnet in
. Therefore, the result obtained as an eigenvalue of the summed operators
will not be the sum
of an eigenvalue of the operator
added linearly to an eigenvalue of the operator
. Indeed, the eigenvalues of
and
are both
, while the eigenvalues of
are
, so a linear relation cannot hold. As Bell points outin [
18], the additivity of expectation values
is a rather unusual property of the quantum states
. The linearity of it is effectuated in quantum mechanics by promoting observable quantities to self-adjoint operators [
25]. It does not hold for the dispersion-free states
of hidden variable theories in general because the eigenvalues of non-commuting observables such as
and
do not add linearly, as we noted above. Consequently, the additivity relation (
11) that holds for quantum states would not hold for the dispersion-free states.
3. Special Case of the Singlet State and EPR-Bohm Observables
Now, the proof of Bell’s famous theorem [
1] is based on Bohm’s spin version of the EPR’s thought experiment [
26], which involves an entangled pair of spin-
particles emerging from a source and moving freely in opposite directions, with particles 1 and 2 subject, respectively, to spin measurements along independently chosen unit directions
and
by Alice and Bob, who are stationed at a spacelike separated distance from each other (see
Figure 1). If initially the pair has vanishing total spin, then the quantum mechanical state of the system is described by the entangled singlet state
where
is a unit vector in arbitrary direction in
and the eigenvalue equation
defines quantum mechanical eigenstates in which the two fermions have spins “up” or “down” in the units of
, with
being the Pauli spin “vector”
. Once the state (
14) is prepared, the observable
of interest is
whose possible eigenvalues, written in terms of the dispersion-free state
instead of the quantum state (
14), are
where
and
are the results of spin measurements made jointly by Alice and Bob along their randomly chosen detector directions
and
. In the singlet state (
14), the joint observable (
16) predicts sinusoidal correlations
between the values of the spins observed about the freely chosen contexts
and
[
6].
For
locally contextual hidden variable theories there is a further requirement that the results of local measurements must be describable by functions that respect local causality, as first envisaged by Einstein [
2] and later formulated mathematically by Bell [
1]. It can be satisfied by requiring that the eigenvalue
of the observable
in (
16) representing the joint result
is factorizable as
, or in Bell’s notation as
with the factorized functions
and
satisfying the following condition of local causality:
Apart from the hidden variables
, the result
of Alice depends
only on the measurement context
, chosen freely by Alice, regardless of Bob’s actions [
1]. And, likewise, apart from the hidden variables
, the result
of Bob depends
only on the measurement context
, chosen freely by Bob, regardless of Alice’s actions. In particular, the function
does not depend on
or
and the function
does not depend on
or
. Moreover, the hidden variables
do not depend on either
,
,
, or
[
11].
The expectation value
of the joint results in the dispersion-free state
should then satisfy the condition
where the hidden variables
originate from a source located in the overlap of the backward light cones of Alice and Bob, and the normalized probability distribution
is assumed to remain statistically independent of the contexts
and
so that
, which is a reasonable assumption. In fact, relaxing this assumption to allow
to depend on
and
introduces a form of non-locality, as explained by Clauser and Horne in footnote 13 of [
27]. Then, since
and
, their product
, setting the following bounds on
:
These bounds are respected not only by local hidden variable theories but also by quantum mechanics and experiments.
5. Additivity of Expectation Values (23) Is An Unjustified Assumption, Equivalent to the Thesis to Be Proven
The key step that led us to the bounds of
on (
21) that are more restrictive than
is the step (
23) of the linear additivity of expectation values. In what follows, I will demonstrate that this step is, in fact, an unjustified assumption, equivalent to the main thesis of the theorem to be proven, just as it is in von Neumann’s now discredited theorem [
9,
18,
25,
28,
29]. But this fact is obscured by the seemingly innocuous built-in linear additivity of integrals used in step (
23). However, as we noted around (
13) and will be further demonstrated in
Section 7, the built-in linear additivity of integrals is physically meaningful only for simultaneously measurable or commuting observables [
25,
28]. It is, therefore, not legitimate to invoke it at step (
23)
without proof. Step (
23) would be valid also in classical physics in which the value of a sum of observable quantities would be the same as the sum of the values each quantity would take separately, because, unlike in quantum mechanics, they would all be simultaneously measurable, yielding only sharp values. Perhaps for this reason it is usually not viewed as an assumption but mistaken for a benign mathematical step. It is also sometimes claimed to be necessitated by Einstein’s requirement of realism [
2]. But I will soon explain why it is a much overlooked unjustified assumption, and demonstrate in
Section 7 that, far from being required by realism, the right-hand side of step (
23), in fact,
contradicts realism, which requires that
every observable of a physical system,including any sums of observables, must be assigned a correct eigenvalue, quantifying one of its preexisting properties.
Moreover, realism has already been adequately accommodated by the very definition of the local functions
and
and their counterfactual juxtaposition on the left-hand side of (
23), as contextually existing properties of the system. Evidently, while a result in only one of the four expectation values corresponding to a sub-experiment that appears on the left-hand side of (
23) can be realized in a given run of a Bell-test experiment, the remaining three results appearing on that side are realizable at least counterfactually, thus fulfilling the requirement of realism [
9]. Therefore, the requirement of realism does not necessitate the left-hand side of (
23) to be equated with its right-hand side in the derivation of (
26). Realism requires definite results
to exist as eigenvalues only counterfactually,
not allfour at once, as they are written on the right-hand side of (
23). What is more, as we will soon see, realism implicit in the prescription (
10) requires the quantity (
24) to be a
correct eigenvalue of the summed operator (
40), but it is not.
On the other hand, given the assumption
of statistical independence and the additivity property of anti-derivatives, mathematically the equality (
23) follows at once because of the linearity built into the integrals, provided we adopt a double standard for additivity: we reject (
23) for von Neumann’s theorem as Bell did in [
18], but accept it unreservedly for Bell’s theorem [
9,
25]. The binary properties of the functions
and
then immediately lead us to the bounds of
on (
21). But, as we saw above, assuming the bounds of
on (
21) leads, conversely, to the assumption (
23) of additivity of expectation values. Thus, assuming the additivity of expectation values (
23) is mathematically equivalent to assuming the bounds of
on the Bell-CHSH sum (
21). In other words, Bell’s argument presented in
Section 4 assumes its conclusion (
26) in the guise of assumption (
23), by implicitly assuming that the expectation functions
determining the eigenvalues
of
are
linear [
29]:
But, as explained by Bohm and Bub in [
29] (see
Appendix B below), this assumed linearity of
is unreasonably restrictive for dispersion-free states
, because the observables defined in (
16) are not simultaneously measurable. However, it allows us to reduce the following correct relation within quantum mechanics
as well as hidden variable theories,
to the relation
which is the same as assumption (
23), albeit written in a more general notation. The equality (
31), on the other hand, is equivalent to the quantum mechanical relation (
33) discussed below, which can be verified using the prescription (
10).The same equality (
31) is also valid for hidden variable theories, because it does not make the mistake of relying on the linearity assumption (
30). This can be verified also using (
10) and the ansatz (
3). Thus, the innocuous-looking linear additivity of integrals in assumption (
23), while mathematically correct, is neither innocent nor physically reasonable.
It is not difficult to understand why appealing to the built-in linear additivity of anti-derivatives is not as innocent or physically reasonable as it may seem. In fact, for non-commuting observables that are not simultaneously measurable, justification of (
23) or (
32) by appealing to the built-in linear additivity of integrals leads to
incorrect equality between unequal physical quantities. The reasons for this were recognized by Grete Hermann [
25] some three decades before the formulation of Bell’s theorem [
1], as part of her insightful criticism of von Neumann’s alleged theorem [
3,
28]. As she explained in [
25], we are not concerned here with classical physics in which all observable quantities are simultaneously measurable yielding only sharp values, and therefore the value of a sum of observable quantities is nothing other than the sum of the values each of those quantities would separately take. Consequently, in classical physics, the averages of such values over individual initial states
of the system can also be meaningfully added linearly, just as assumed in step (
23) or (
32), because there is no scope for any contradiction between the averages obtained by evaluating the left-hand side and the right-hand side of these equations. Therefore, in classical physics linear additivity of expectation values remains consistent with the built-in linear additivity of anti-derivatives. However, the same cannot be assumed without proof for the dispersion-free states
of hidden variable theories, because, in that case, the values of the observable quantities are eigenvalues of the corresponding quantum mechanical operators dictated by the ansatz (
3), and, as we noted above and toward the end of
Section 2, the eigenvalue
of the summed observable
is not equal to the sum
of the eigenvalues
of
, unless the observables
constituting the sum
are simultaneously measurable.
Thus, an important step in the proof of (26) is missing. A necessary step that would prove the consistency of the built-in linear additivity of anti-derivatives with the non-additivity of expectation values for the
non-commuting observables. In Equation (
43) of
Section 7 below we will see the difference between the eigenvalue of the summed operator and the sum of individual eigenvalues explicitly. It will demonstrate how, in hidden variable theories Equation (
23) or (
32) involving averages of eigenvalues ends up equating unequal averages of physical quantities in general. It will thereby prove that, while valid in classical physics and for simultaneously measurable observables, Equation (
23) or (
32) is
not valid for hidden variable theories in general. Insisting otherwise thus amounts to
assuming the validity of this equation
without proof, despite the contrary evidence just presented [
25]. That, in turn, amounts to assuming the very thesis to be proven — namely, the bounds of
on the Bell-CHSH sum (
21). Consequently, the onlycorrect meaning assignable to (
23) or (
26) is that it is valid only in classical physics and/or for commuting observables.
Sometimes assumption (
23) is justified on statistical grounds. It is argued that the four sub-experiments appearing on the left-hand side of (
23) with different experimental settings
,
,
etc. can be performed independently of each other, on possibly different occasions, and then the resulting averages are added together at a later time for statistical analysis. If the number of experimental runs for each pair of settings is sufficiently large, then, theoretically, the sum of the four averages appearing on the left-hand side of (
23) are found not to exceed the bounds of
, thus justifying the equality (
23). This can be easily verified in numerical simulations (see Ref. [27] cited in [
13]). However,this heuristic argument is not an analytical proof of the bounds. What it implicitly neglects to take into account by explicitly assuming that the four sub-experiments can be performed independently, is that the sub-experiments involve mutually exclusive pairs of settings such as
and
in physical space, and thus involve non-commuting observables that cannot be measured simultaneously [
9]. Unless the statistical analysis takes this physical fact into account, it cannot be claimed to have any relevance for the Bell-test experiments [
16]. For ignoring this physical fact amounts to incorrectly assuming that the spin observables
,
etc. are mutually commuting, and thus simultaneously measurable, for which assumption (
23) is indeed valid, as demonstrated below in
Section 7 (see the discussion around (
46)). On the other hand, when the non-commutativity of the observables involved in the sub-experiments is taken into account in numerical simulations, the bounds on (
21) turn out to be
, as shown in [
10,
11] and Ref. [27] cited in [
13]. In other words, such a statistical argument is simply assumption (
23) in disguise.
Another important point to recognize here is that the above derivation of the stringent bounds of
on (
21) for a locally causal dispersion-free counterpart
of the quantum mechanical singlet state (
14) must comply with the heuristics of the contextual hidden variable theories we discussed in
Section 2. If it does not, then the bounds of
cannot be claimed to have any relevance for the viability of local hidden variable theories [
24]. Therefore, as discussed in
Section 2, in a contextual hidden variable theory all of the observables
of any physical system,
including their sum
, which also represents a physical quantity in the Hilbert space formulation of quantum mechanics [
3] whether or not it is observed, must be assigned unique eigenvalues
and
, respectively, in the dispersion-free states
of the system, regardless of whether these observables are simultaneously measurable. In particular, while the summed observable (
40) discussed below is never observed in the Bell-test experiments, realism nevertheless requires it to be assigned a unique eigenvalue in accordance with the ansatz (
3) and the prescription (
10).
7. Additivity of Expectation Values does not Hold for Dispersion-Free States
The problem with Equation (
38) is that, while the joint results
,
etc. appearing on the left-hand side of Equation (
23) are possible eigenvalues of the products of spin operators
,
etc., their summation
appearing as the integrand on the right-hand side of Equation (
38) or (
23) is
not an eigenvalue of the summed operator
because the spin operators
and
,
etc., and therefore
,
etc., do not commute with each other:
Consequently, Equation (
38) would hold within any hidden variable theory
only if the operators
,
etc. were commuting operators. As we discussed, this is well known from the famous criticisms of von Neumann’s theorem against hidden variable theories [
9,
18,
25,
28]. While the equality (
23) of the sum of expectation values with the expectation value of the sum is respected in quantum mechanics, it does not hold for hidden variable theories [
18]. Nor does local realism necessitate the linear additivity (
30) of eigenvalues for individual dispersion-free states
.
This problem, however, suggests its own resolution. We can work out the correct eigenvalue
of the summed operator (
40), at least formally, as I have worked out in
Appendix C below. The correct version of Equation (
38) is then
where
is the correct eigenvalue of the summed operator (
40), with its non-commuting part separated out as the operator
Here
in general, because the vector
does not vanish in general. It works out to be
The details of how this separation is accomplished using (
41) can be found in
Appendix C below. From (
43), it is now easy to appreciate that the additivity of expectation values (
23) assumed by Bell can hold only if the expectation value
of the non-commuting part within the eigenvalue
of the summed operator (
40) is zero. But that is possible only if the operators
,
etc. constituting the sum (
40) commute with eachother. In general, if the operators
,
etc. in (
40) do not commute with each other, then we would have
But the operators
,
etc. indeed do not commute with each other, because the pairs of directions
,
etc. in (
40) are mutually exclusive directions in
. Therefore, the additivity of expectation values assumed at step (
23) in the derivation of (
26) is unjustifiable. Far from being necessitated by realism, it actually contradicts realism.
Since three of the four results appearing in the expression (
39) can be realized only counterfactually, their summation in (
39) cannot be realized
even counterfactually [
9]. Thus, in addition to not being a correct eigenvalue of the summed operator (
40) as required by the prescription (
10) for hidden variable theories, the quantity appearing in (
39) is, in fact, an entirely fictitious quantity, with no counterpart in any possible world, apart from in the trivial case when all observables are commutative. By contrast, the correct eigenvalue (
43) of the summed operator (
40) can be realized at least counterfactually because it is a genuine eigenvalue of that operator, thereby satisfying the requirement of realism correctly, in accordance with the prescription (
10) for hidden variable theories. Using (
43), all five of the observables appearing on both sides of the quantum mechanical Equation (
33) can be assigned unique and correct eigenvalues [
9].
Once this oversight is ameliorated, it is not difficult to show that the conclusion of Bell’s theorem no longer follows. For then, using the correct eigenvalue (
43) of (
40) instead of (
39) on the right-hand side of (
23), we have the equation
instead of (
23), which implements local realism correctly on both of its sides, as required by the prescription (
10) we discussed in
Section 2. This Equation (
47) is thus the correct dispersion-free counterpart of the equivalence (
33) for the quantum mechanical expectation values [
9]. It can reduce to Bell’s assumption (
23) only when the expectation value
of the non-commuting part within the eigenvalue
of the summed operator (
40) happens to be vanishing. It thus expresses the correct relationship (
31) among the expectation values for the singlet state (
14) in the local hidden variable framework considered by Bell [
1]. Recall again from the end of
Section 2 that the quantum mechanical relation (
33) is an unusual property of the quantum states
. As Bell stressed in [
18], “[t]here is no reason to demand it individually of the hypothetical dispersion free states, whose function it is to reproduce the
measurable peculiarities of quantum mechanics
when averaged over.” Moreover, in Section V of [
9] I have demonstrated that the bounds on the right-hand side of (
47) are
instead of
. An alternative derivation of these bounds follows from the magnitude
of the vector defined in (
45), which, as proved in
Appendix D below, is bounded by 2, and thereforethe eigenvalue
of the operator (
44) obtained as its expectation value
is bounded by
, giving
Substituting these into (
43), together with the bounds of
we worked out before on the commuting part (
39), gives
which is constrained to be real despite the square root in the expression (
43) because the operator (
40) is Hermitian. Consequently, we obtain the following Tsirel’son’s bounds in the dispersion-free state, on the right-hand side of (
47):
Given the correct relation (
47) between expectation values instead of the flawed assumption (
23), we thus arrive at
Since the bounds of
we have derived on the Bell-CHSH sum of expectation values are the same as those predicted by quantum mechanics and observed in the Bell-test experiments, the conclusion of Bell’s theorem is mitigated. What is ruled out by these experiments is not local realism but the assumption of the additivity of expectation values, which does not hold for non-commuting observables in dispersion-free states of any hidden variable theories to begin with.
It is also instructive to note that the intermediate bounds
on the Bell-CHSH sum (
21), instead of the extreme bounds
or
, follow in the above derivation of (
51) as a consequence of the geometry of physical space [
5,
6]. Thus, what is brought out in it is the oversight of the non-commutative or Clifford-algebraic attributes of the physical space in Bell’s derivation of the bounds
in (
26). Indeed, it is evident from
Appendix D below that the geometry of physical space imposes the bounds
on the magnitude of the vector (
45), which, in turn, lead us to the bounds
in (
51). This is in sharp contrast with the traditional view of these bounds as due to non-local influences,stemming from a failure of the locality condition (
18). But in the derivation of (
51) above, the condition (
18) is strictly respected. Therefore, the strength of the bounds
in (
51) is a consequence —
not of non-locality or non-reality, but of the geometry of physical space [
5,
6]. Non-locality or non-reality is necessitated only if one erroneously insists on linear additivity (
23) of eigenvalues of non-commuting observables for each
individual dispersion-free state
.