How can we construct a quantum theory, unencumbered by history and all the false starts that preceded the development of the present-day Standard Model of particle physics? Well... here is one way. I shall begin with a nonrelativistic quantum theory of a single-particle harmonic oscillator. I then show how the same theory can be formulated relativistically. I then move on to the corresponding quantum field theory (the formalism of which, surprisingly, is almost identical in the nonrelativistic vs. the relativistic case.)

Nonrelativistic single-particle harmonic oscillator

The classical Lagrangian of a single-particle harmonic oscillator in an external potential (OK, technically if it's in an external potential, it's not really harmonic anymore, but let us not get bogged down in semantics) is given by:

$$L=\frac{1}{2}m\dot{\vec{q}}^2-\frac{1}{2}k\vec{q}^2-V(\vec{q}),$$

where the particle's mass is given by $m$, its vector position is $\vec{q}$, and $k$ (with the dimensions of force over length) would be a spring constant. The corresponding Euler-Lagrange equation, the momentum and the Hamiltonian are given, respectively, by:

\begin{align*}\frac{\partial L}{\partial\vec{q}}-\frac{d}{dt}\frac{\partial L}{\partial\dot{\vec{q}}}&=-m\ddot{\vec{q}}-k\vec{q}-\vec{\nabla}V(\vec{q})=0,\\
\vec{p}&=\frac{\partial L}{\partial\dot{\vec{q}}}=m\dot{\vec{q}},\\
H&=\vec{p}\cdot\dot{\vec{q}}-L=\frac{\vec{p}^2}{2m}+\frac{1}{2}k\vec{q}^2+V(\vec{q}),
\end{align*}

This is the classical theory. To begin the transition to the quantum theory, we move all terms in the definition of the Hamiltonian to the left-hand side, and then multiply the equation by the term $\psi=e^{i(\vec{p}\cdot\vec{q}-Ht)/\hbar}$ where $\hbar$ is a constant with the dimensions of action (energy times time)::

$$\left[H-\frac{\vec{p}^2}{2m}-\frac{1}{2}k\vec{q}^2-V(\vec{q})\right]e^{i(\vec{p}\cdot\vec{q}-Ht)/\hbar}=0.$$

Note that this is still the classical theory: we just performed the algebraically trivial action of multiplying both sides of an equation by a manifestly nonzero term. But now notice that $\vec{p}\psi=-i\hbar\vec{\nabla}\psi$. Similarly, $H\psi=i\hbar\partial_t\psi$. That is, $\vec{p}$ and $H$ are eigenvalues of the operators $-i\hbar\vec{\nabla}$ and $i\hbar\partial_t$, respectively, with the corresponding eigenfunction being $\psi$. This allows us to rewrite the preceding equation in the form,

$$\left[i\hbar\partial_t+\frac{\hbar^2}{2m}\vec{\nabla}^2-\frac{1}{2}k\vec{q}^2-V(\vec{q})\right]e^{i(\vec{p}\cdot\vec{q}-Ht)/\hbar}=0.$$

It is important to emphasize that this is still the classical theory! Up to this point, everything we did was essentially trivial algebra and a little bit of calculus. No new assumptions were introduced.

But now things are about to get interesting. Suppose we try to solve the equation, $\left[i\hbar\partial_t+\frac{\hbar^2}{2m}\vec{\nabla}^2-\frac{1}{2}k\vec{q}^2-V(\vec{q})\right]\psi=0.$ We get solutions that are eigenfunctions of the two operators $-i\hbar\vec{\nabla}$ and $i\hbar\partial_t$. On account of this equation being homogeneous, if $\psi$ is a solution then $\alpha\psi$ is also a solution; and given two solutions $\psi_1$ and $\psi_2$, their sum $\psi_1+\psi_2$ is also a solution... but it is not an eigenfunction anymore.

This is where the switch the quantum physics takes place: when we assert that reality is described not by the eigenvalues $\vec{p}$ and $H$, but by the state represented by the linear combinations of $\psi$-s. This is a much richer space of solutions that includes all the classical solutions but also solutions with no classical equivalent.

So how and when can the classical solution be recovered? Given an eigenfunction $\psi$ that is normalized ($\psi\psi^\star=1$), the classical value of the momentum can be recovered as: $\vec{p}=\psi^\star\vec{\nabla}\psi$. When $\psi$ is not a normalized eigenfunction, the value thus obtained is called the expectation value of $\psi$ with respect to the operator $\vec{\nabla}$.

Let us now do a little bit of straightforward algebra:

\begin{align*}
\left[i\hbar\partial_t-\left(\frac{\hbar}{\sqrt{2m}}\vec{\nabla}+\sqrt{\frac{k}{2}}\vec{q}\right)\left(\frac{-\hbar}{\sqrt{2m}}\vec{\nabla}+\sqrt{\frac{k}{2}}\vec{q}\right)+\hbar\sqrt{\frac{k}{4m}}\left(\vec{\nabla}\cdot\vec{q}-\vec{q}\cdot\vec{\nabla}\right)-V(\vec{q})\right]e^{i(\vec{p}\cdot\vec{q}-Ht)/\hbar}&=0,\\
\left[i\hbar\partial_t-\left(\frac{\hbar}{\sqrt{2m}}\vec{\nabla}+\sqrt{\frac{k}{2}}\vec{q}\right)\left(\frac{-\hbar}{\sqrt{2m}}\vec{\nabla}+\sqrt{\frac{k}{2}}\vec{q}\right)+\frac{\hbar}{2}\sqrt{\frac{k}{m}}-V(\vec{q})\right]e^{i(\vec{p}\cdot\vec{q}-Ht)/\hbar}&=0.
\end{align*}

Now let us introduce the symbols

\begin{align*}
\omega&=\sqrt{\frac{k}{m}},\\
\hat{a}^\dagger&=\frac{1}{\sqrt{2\hbar\omega}}\left(\frac{-\hbar}{\sqrt{m}}\vec{\nabla}+\sqrt{k}\vec{q}\right),\\
\hat{a}&=\frac{1}{\sqrt{2\hbar\omega}}\left(\frac{\hbar}{\sqrt{m}}\vec{\nabla}+\sqrt{k}\vec{q}\right).
\end{align*}

Defining the commutator $[x,y]=xy-yx$, we notice that

$$[\hat{a},\hat{a}^\dagger]=\hat{a}\hat{a}^\dagger-\hat{a}^\dagger\hat{a}=\frac{1}{\hbar\omega}\left(\hbar\sqrt{\frac{k}{m}}\vec{\nabla}\cdot\vec{q}-\hbar\sqrt{\frac{k}{m}}\vec{q}\cdot\vec{\nabla}\right)=1.$$

Using $\omega$, $\hat{a}$ and $\hat{a}^\dagger$, we can rewrite the previous equation as

$$\left[i\hbar\partial_t-\hbar\omega\left(\hat{a}\hat{a}^\dagger-\frac{1}{2}\right)-V(\vec{q})\right]e^{i(\vec{p}\cdot\vec{q}-Ht)/\hbar}=0,$$

or, after reintroducing $\hat{H}=i\hbar\partial_t$ and rearranging,

$$\hat{H}=\hbar\omega\left(\hat{a}^\dagger\hat{a}+\frac{1}{2}\right)+V(\vec{q}).$$

 We can also form the commutators

\begin{align*}
[\hat{H},\hat{a}]&=\hbar\omega(\hat{a}\hat{a}^\dagger\hat{a}-\hat{a}\hat{a}\hat{a}^\dagger)=\hbar\omega\hat{a}(\hat{a}^\dagger\hat{a}-\hat{a}\hat{a}^\dagger)=-\hbar\omega\hat{a},\\
[\hat{H},\hat{a}^\dagger]&=\hbar\omega(\hat{a}\hat{a}^\dagger\hat{a}^\dagger-\hat{a}^\dagger\hat{a}\hat{a}^\dagger)=h\omega(\hat{a}\hat{a}^\dagger-\hat{a}^\dagger\hat{a})\hat{a}^\dagger=\hbar\omega\hat{a}^\dagger,\\
\end{align*}

and, using these commutators and the definition of the $n$-th eigenvalue of $\hat{H}$ as

$$\hat{H}\psi_n=E_n\psi_n,$$

we find that

\begin{align*}
\hat{H}\hat{a}\psi_n&=\left([\hat{H},\hat{a}]+\hat{a}\hat{H}\right)\psi_n=(E_n-\hbar\omega)\hat{a}\psi_n,\\
\hat{H}\hat{a}^\dagger\psi_n&=\left([\hat{H},\hat{a}^\dagger]+\hat{a}^\dagger\hat{H}\right)\psi_n=(E_n+\hbar\omega)\hat{a}^\dagger\psi_n.
\end{align*}

Moreover, if we assume that there exists a $\psi_0$ such that $\hat{a}\psi_0=0$, we find that

\begin{align*}
\hbar\omega\hat{a}^\dagger\hat{a}\psi_0&=0,\\
\hat{H}\psi_0&=\frac{\hbar\omega}{2}\psi_0,\\
E_n&=\left(n+\frac{1}{2}\right)\hbar\omega,\\
\hat{a}^\dagger\hat{a}\psi_n&=n\psi_n,\\
\end{align*}

and therefore,

\begin{align*}
\hat{a}\psi_n&=\sqrt{n}\psi_{n-1},\\
\hat{a}^\dagger\psi_{n-1}&=\sqrt{n}\psi_n,
\end{align*}

which implies that the "annihilation" and "creation" operators $\hat{a}$ and $\hat{a}^\dagger$ can be represented as

\begin{align*}
\hat{a}=\begin{pmatrix}0&0&0&0&\ldots\\\sqrt{1}&0&0&0&\ldots\\0&\sqrt{2}&0&0&\ldots\\0&0&\sqrt{3}&0&\ldots\\\vdots&\vdots&\vdots&\vdots&\ddots\end{pmatrix},&\qquad\qquad&
\hat{a}^\dagger=\begin{pmatrix}0&\sqrt{1}&0&0&\ldots\\0&0&\sqrt{2}&0&\ldots\\0&0&0&\sqrt{3}&\ldots\\0&0&0&0&\ldots\\\vdots&\vdots&\vdots&\vdots&\ddots\end{pmatrix}.
\end{align*}

So by introducing the sole non-classical assumption that linear combinations of eigenfunctions also represent physical states, we arrived at the conclusion that the quantum harmonic oscillator can be represented using creation and annihilation operators that create and destroy eigenstates.

The relativistic harmonic oscillator

The logic in the relativistic case is very similar (putting aside the known issues involving relativistic quantum particle physics, such as causality violations or negative probabilities.) We start with the Lagrangian

$$L=-\sqrt{u_\alpha u^\alpha}\left(\textstyle\frac{1}{2}kq_iq^i+mc^2\right)\frac{d\tau}{dt}-V(\vec{q}),$$

where $u_\alpha$ is now the four-velocity of the object, $c$ is the speed of light, while $\tau$ is proper time. A little bit of algebraic manipulation leads to the Euler-Lagrange equation:

\begin{align*}
L&=-\sqrt{\frac{c^2-\dot{\vec{q}}^2}{c^2-\dot{\vec{q}}^2}}\left(\textstyle\frac{1}{2}k\vec{q}^2+mc^2\right)\frac{\sqrt{c^2-\dot{\vec{q}}^2}}{c}-V(\vec{q})\\
&=-\sqrt{1-\frac{\dot{\vec{q}}^2}{c^2}}\left(\textstyle\frac{1}{2}k\vec{q}^2+mc^2\right)-V(\vec{q}),\\\frac{\partial L}{\partial\vec{q}}-\frac{d}{dt}\frac{\partial L}{\partial\dot{\vec{q}}}&=-\frac{\frac{1}{2}k\vec{q}^2+mc^2}{c^2\left(1-\displaystyle\frac{\dot{\vec{q}}^2}{c^2}\right)^{3/2}}\ddot{\vec{q}}-\frac{k\vec{q}}{\sqrt{1-\displaystyle\frac{\dot{\vec{q}}^2}{c^2}}}-\vec{\nabla}V(\vec{q}).
\end{align*}

Given

$$\vec{p}=\frac{\partial L}{\partial\dot{\vec{q}}}=\frac{\left(\frac{1}{2}k\vec{q}^2+mc^2\right)\dot{\vec{q}}}{c^2\sqrt{1-\displaystyle\frac{\dot{\vec{q}}^2}{c^2}}},$$

and

$$\dot{\vec{q}}=\frac{c^2\vec{p}}{\sqrt{c^2\vec{p}^2+\left(\frac{1}{2}k\vec{q}^2+mc^2\right)^2}},$$

which leads to

$$\sqrt{1-\frac{\dot{\vec{q}}^2}{c^2}}=\frac{1}{\sqrt{1+\displaystyle\frac{c^2\vec{p}^2}{\left(\frac{1}{2}k\vec{q}^2+mc^2\right)^2}}},$$

we obtain the Hamiltonian

$$H=\vec{p}\cdot\dot{\vec{q}}-L=\sqrt{\left(\textstyle\frac{1}{2}k\vec{q}^2+mc^2\right)^2+\vec{p}^2c^2}+V(\vec{q}),$$

or

$$(H-V)^2=\left(\textstyle\frac{1}{2}k\vec{q}^2+mc^2\right)^2+\vec{p}^2c^2,$$

or

\begin{align*}
\left[(H-V)^2-\left(\textstyle\frac{1}{2}k\vec{q}^2+mc^2\right)^2-\vec{p}^2c^2\right]e^{i[\vec{p}\cdot\vec{q}-Ht]/\hbar}&=0,\\
\left[(\hat{H}-V)^2-\left(\textstyle\frac{1}{2}k\vec{q}^2+mc^2\right)^2+\hbar^2c^2\vec{\nabla}^2\right]e^{i[\vec{p}\cdot\vec{q}-Ht]/\hbar}&=0,\\
\left[(\hat{H}-V)^2-\textstyle\frac{1}{4}k^2\vec{q}^4-k\vec{q}^2mc^2-m^2c^4+\hbar^2c^2\vec{\nabla}^2\right]e^{i[\vec{p}\cdot\vec{q}-Ht]/\hbar}&=0.
\end{align*}

If the spring constant is small enough, such that $\frac{1}{2}k\vec{q}^2\ll mc^2$, we get

\begin{align*}
\left[(\hat{H}-V)^2-m^2c^4-kmc^2\vec{q}^2+\hbar^2c^2\vec{\nabla}^2\right]e^{i[\vec{p}\cdot\vec{q}-Ht]/\hbar}&=0,\\
\left[(\hat{H}-V)^2-m^2c^4-2mc^2\left(\frac{k}{2}\vec{q}^2-\frac{\hbar^2}{2m}\vec{\nabla}^2\right)\right]e^{i[\vec{p}\cdot\vec{q}-Ht]/\hbar}&=0,\\
\left[(\hat{H}-V)^2-m^2c^4-2mc^2\left(\sqrt{\frac{k}{2}}\vec{q}+\frac{\hbar}{\sqrt{2m}}\vec{\nabla}\right)\left(\sqrt{\frac{k}{2}}\vec{q}-\frac{\hbar}{\sqrt{2m}}\vec{\nabla}\right)+\hbar mc^2\sqrt{\frac{k}{m}}\right]e^{i[\vec{p}\cdot\vec{q}-Ht]/\hbar}&=0,\\
\left[(\hat{H}-V)^2-m^2c^4-2\hbar\omega mc^2\left(\hat{a}\hat{a}^\dagger-\frac{1}{2}\right)\right]e^{i[\vec{p}\cdot\vec{q}-Ht]/\hbar}&=0,
\end{align*}

or

$$\hat{H}=V(\vec{q})+mc^2\sqrt{1+\frac{2}{mc^2}\hbar\omega\left(\hat{a}\hat{a}^\dagger-\frac{1}{2}\right)}\simeq V(\vec{q})\pm\left[mc^2+\hbar\omega\left(\hat{a}^\dagger\hat{a}+\frac{1}{2}\right)\right].$$

So once again we get creation and annihilation operators; and in the nonrelativistic approximation, we get back the nonrelativistic result, apart from the nonessential constant term $mc^2$. Neat! Except for the negative energy modes. One way to deal with that problem is to move from a particle to a field theory.

The field theory

So how do we build a field theory? We obviously need a Lagrangian density. Inspired by the harmonic oscillator, and using coefficients to ensure that the Lagrangian density does, indeed, have the dimensions of energy density, we can write

$${\cal L}=\frac{1}{2}\rho(\partial_t\phi)^2-\frac{1}{2}\rho c^2(\vec{\nabla}\phi)^2-\frac{1}{2}\kappa\phi^2-V(\phi),$$

where $\rho$ has the dimensions of mass density, $\kappa$ has the dimensions of force density over length, and the unknown scalar field $\phi$ has the dimensions of length. (Alternatively, we could absorb the coupling constant $\rho$ into the field definition. In natural units, this would mean that $\phi$ would have the dimensions of energy and the Lagrangian would have the dimensions of energy density, i.e., $E^4$; also, $\kappa$ would have the dimensions of $E^2.$) The velocity $c$ need not be the speed of light in the case of a nonrelativistic theory; if the theory is relativistic, however, $c$ is the vacuum speed of light. (Otherwise, curiously, the nonrelativistic and the relativistic versions are identical.)

The Euler-Lagrange equation is given by

$$\frac{\partial{\cal L}}{\partial\phi}-\frac{\partial}{\partial t}\frac{\partial{\cal L}}{\partial(\partial_t\phi)}-\vec{\nabla}\frac{\partial{\cal L}}{\partial\vec{\nabla}\phi}=-\rho\partial_t^2\phi-\rho c^2\vec{\nabla}^2\phi-\kappa\phi-V'(\phi)=0,$$

whereas the momentum density $\pi$ and, subsequently, the Hamiltonian density ${\cal H}$, are given by

\begin{align*}
\pi&=\frac{\partial{\cal L}}{\partial(\partial_t\phi)}=\rho\partial_t\phi,\\
{\cal H}&=\pi\partial_t\phi-{\cal L}=\frac{\pi^2}{2\rho}+\frac{1}{2}\rho c^2(\vec{\nabla}\phi)^2+\frac{1}{2}\kappa\phi^2+V(\phi).
\end{align*}

We now define

$$\omega=\sqrt{c^2\vec{k}^2+\frac{\kappa}{\rho}}.$$

Next, we write $\phi$ in the form of a Fourier integral:

\begin{align*}
\phi(t,\vec{x})&=\int\frac{d^3\vec{k}}{(2\pi)^3\sqrt{2\hbar\omega}}\frac{\hbar}{\sqrt{\rho}}\left[a(\omega,\vec{k})e^{-i[\omega t-\vec{k}\cdot\vec{x}]}+a^\dagger(\omega,\vec{k})e^{i[\omega t-\vec{k}\cdot\vec{x}]}\right]\\
&=\int\frac{d^3\vec{k}}{(2\pi)^3\sqrt{2\hbar\omega}}\frac{\hbar}{\sqrt{\rho}}\left[a(\omega,\vec{k})+a^\dagger(-\omega,-\vec{k})\right]e^{-i[\omega t-\vec{k}\cdot\vec{x}]}.
\end{align*}

Differentiating $\phi$, we obtain expressions for $\pi$ and $\vec{\nabla}\phi$:

\begin{align*}
\pi(t,\vec{x})&=\int\frac{d^3\vec{k}}{(2\pi)^3\sqrt{2\hbar\omega}}\hbar\sqrt{\rho}(-i\omega)\left[a(\omega,\vec{k})e^{-i[\omega t-\vec{k}\cdot\vec{x}]}-a^\dagger(\omega,\vec{k})e^{i[\omega t-\vec{k}\cdot\vec{x}]}\right]\\
&=\int\frac{d^3\vec{k}}{(2\pi)^3\sqrt{2\hbar\omega}}\hbar\sqrt{\rho}(-i\omega)\left[a(\omega,\vec{k})-a^\dagger(-\omega,-\vec{k})\right]e^{-i[\omega t-\vec{k}\cdot\vec{x}]},\\
\vec{\nabla}\phi(t,\vec{x})&=\int\frac{d^3\vec{k}}{(2\pi)^3\sqrt{2\hbar\omega}}\frac{\hbar}{\sqrt{\rho}}(i\vec{k})\left[a(\omega,\vec{k})e^{-i[\omega t-\vec{k}\cdot\vec{x}]}+a^\dagger(\omega,\vec{k})e^{i[\omega t-\vec{k}\cdot\vec{x}]}\right]\\
&=\int\frac{d^3\vec{k}}{(2\pi)^3\sqrt{2\hbar\omega}}\frac{\hbar}{\sqrt{\rho}}(i\vec{k})\left[a(\omega,\vec{k})+a^\dagger(-\omega,-\vec{k})\right]e^{-i[\omega t-\vec{k}\cdot\vec{x}]}.
\end{align*}

Again, everything up to this point has been the classical theory. We just performed some calculus and some algebraic manipulation, that's all.

To transition to the quantum theory, we next assume that the Fourier coefficients $a$ and $a^\dagger$ are in fact eigenvalues of corresponding operators $\hat{a}$ and $\hat{a}^\dagger$, acting on an eigenfunction $\psi$:

\begin{align*}
a\psi&=\hat{a}\psi,\\
a^\dagger\psi&=\hat{a}^\dagger\psi,\\
\end{align*}

Furthermore, we assume that these operators have the commutator

$$[\hat{a}(\omega,\vec{k}),\hat{a}^\dagger(\omega,\vec{k}')]=(2\pi)^3\delta^3(\vec{k}-\vec{k}').$$

We can use this expression to write the commutator of $\phi$ and $\pi$:

\begin{align*}
[\hat{\phi}(t,\vec{x}),\hat{\pi}(t,\vec{x}')]&=-\frac{i\hbar}{2}\int\int\frac{d^3\vec{k}d^3\vec{k}'}{(2\pi)^6}\Big[\hat{a}^\dagger(\omega,\vec{k})\hat{a}(\omega,\vec{k}')e^{i[\omega t-\vec{k}\cdot\vec{x}]}e^{-i[\omega t-\vec{k}'\cdot\vec{x}']}\\
&\hskip 1.5in {} - \hat{a}(\omega,\vec{k}')\hat{a}^\dagger(\omega,\vec{k})e^{i[\omega t-\vec{k}\cdot\vec{x}]}e^{-i[\omega t-\vec{k}'\cdot\vec{x}']}\\
&\hskip 1.5in {} + \hat{a}^\dagger(\omega,\vec{k}')\hat{a}(\omega,\vec{k})e^{-i[\omega t-\vec{k}\cdot\vec{x}]}e^{i[\omega t-\vec{k}'\cdot\vec{x}']}\\
&\hskip 1.5in {} - \hat{a}(\omega,\vec{k})\hat{a}^\dagger(\omega,\vec{k}')e^{-i[\omega t-\vec{k}\cdot\vec{x}]}e^{i[\omega t-\vec{k}'\cdot\vec{x}']}\Big]\\
&=-\frac{i\hbar}{2}\int\int\frac{d^3\vec{k}d^3\vec{k}'}{(2\pi)^3}\left[-\delta^3(\vec{k}-\vec{k}')e^{-i[\vec{k}\cdot\vec{x}-\vec{k}'\cdot\vec{x}']}-\delta^3(\vec{k}-\vec{k}')e^{i[\vec{k}\cdot\vec{x}-\vec{k}'\cdot\vec{x}']}\right]\\
&=-\frac{i\hbar}{2}\int\frac{d^3\vec{k}}{(2\pi)^3}\left[-e^{-i\vec{k}\cdot[\vec{x}-\vec{x}']}-e^{i\vec{k}\cdot[\vec{x}-\vec{x}']}\right]=i\hbar\delta^3(\vec{x}-\vec{x}'),
\end{align*}

where we used the properties and definition of the Dirac delta function to obtain the final expression.

Finally, we can form an expression for the Hamiltonian. First, we note that

\begin{align*}
\kappa\hat{\phi}^2&=\kappa\int\frac{d^3\vec{k}}{(2\pi)^3\sqrt{2\hbar\omega}}\frac{\hbar}{\sqrt{\rho}}\left[\hat{a}(\omega,\vec{k})+\hat{a}^\dagger(-\omega,-\vec{k})\right]e^{-i[\omega t-\vec{k}\cdot\vec{x}]}\\
&~~{}\times\int\frac{d^3\vec{k}'}{(2\pi)^3\sqrt{2\hbar\omega}}\frac{\hbar}{\sqrt{\rho}}\left[\hat{a}(-\omega,-\vec{k}')+\hat{a}^\dagger(\omega,\vec{k}')\right]e^{i[\omega t-\vec{k}'\cdot\vec{x}]}\\
&=\int\int\frac{d^3\vec{k}d^3\vec{k}'}{(2\pi)^6}\frac{\hbar}{2\omega}\frac{\kappa}{\rho}\left[\hat{a}(\omega,\vec{k})+\hat{a}^\dagger(-\omega,-\vec{k})\right]\left[\hat{a}(-\omega,-\vec{k}')+\hat{a}^\dagger(\omega,\vec{k}')\right]e^{i(\vec{k}-\vec{k}')\cdot\vec{x}},\\
\frac{\hat{\pi}^2}{\rho}&=\frac{1}{\rho}\int\frac{d^3\vec{k}}{(2\pi)^3\sqrt{2\hbar\omega}}\hbar\sqrt{\rho}(-i\omega)\left[\hat{a}(\omega,\vec{k})-\hat{a}^\dagger(-\omega,-\vec{k})\right]e^{-i[\omega t-\vec{k}\cdot\vec{x}]}\\
&~~{}\times\int\frac{d^3\vec{k}'}{(2\pi)^3\sqrt{2\hbar\omega}}\hbar\sqrt{\rho}(-i\omega)\left[\hat{a}(-\omega,-\vec{k}')-\hat{a}^\dagger(\omega,\vec{k}')\right]e^{i[\omega t-\vec{k}'\cdot\vec{x}]}\\
&=-\int\int\frac{d^3\vec{k}d^3\vec{k}'}{(2\pi)^6}\frac{\hbar}{2\omega}\omega^2\left[\hat{a}(\omega,\vec{k})-\hat{a}^\dagger(-\omega,-\vec{k})\right]\left[\hat{a}(-\omega,-\vec{k}')-\hat{a}^\dagger(\omega,\vec{k}')\right]e^{i(\vec{k}-\vec{k}')\cdot\vec{x}},\\
\rho c^2\vec{\nabla}\hat{\phi}\cdot\vec{\nabla}\hat{\phi}&=\rho c^2\int\frac{d^3\vec{k}}{(2\pi)^3\sqrt{2\hbar\omega}}\frac{\hbar}{\sqrt{\rho}}(i\vec{k})\left[\hat{a}(\omega,\vec{k})+\hat{a}^\dagger(-\omega,-\vec{k})e\right]^{-i[\omega t-\vec{k}\cdot\vec{x}]}\\
&~~{}\cdot\int-\frac{d^3\vec{k}'}{(2\pi)^3\sqrt{2\hbar\omega}}\frac{\hbar}{\sqrt{\rho}}(i\vec{k}')\left[\hat{a}(-\omega,-\vec{k}')+\hat{a}^\dagger(\omega,\vec{k}')e\right]^{i[\omega t-\vec{k}'\cdot\vec{x}]}\\
&=\int\int\frac{d^3\vec{k}d^3\vec{k}'}{(2\pi)^6}\frac{\hbar}{2\omega}c^2\vec{k}\cdot \vec{k}'\left[\hat{a}(\omega,\vec{k})+\hat{a}^\dagger(-\omega,-\vec{k})\right]\left[\hat{a}(-\omega,-\vec{k}')+\hat{a}^\dagger(\omega,\vec{k}')\right]e^{i(\vec{k}-\vec{k}')\cdot\vec{x}},\\
\end{align*}

Then, the total Hamiltonian (not the Hamiltonian density) can be obtained by utilizing the definition of the Fourier transform and its inverse, as follows:

\begin{align*}
\hat{H}&=\int\hat{\cal H}~d^3x=\int d^3x\left\{\frac{\hat{\pi}^2}{2\rho}+\frac{1}{2}\rho c^2(\vec{\nabla}\hat{\phi})^2+\frac{1}{2}\kappa\hat{\phi}^2+V(\hat{\phi})\right\},\\
&=\int\frac{d^3\vec{k}}{(2\pi)^3}\frac{\hbar}{4\omega}\int\int\frac{d^3\vec{x}d^3\vec{k}'}{(2\pi)^3}\bigg\{\frac{\kappa}{\rho}\left[\hat{a}(\omega,\vec{k})+\hat{a}^\dagger(-\omega,-\vec{k})\right]\left[\hat{a}(-\omega,-\vec{k}')+\hat{a}^\dagger(\omega,\vec{k}')\right]\\
&\hskip 2.2in{}-\omega^2\left[\hat{a}(\omega,\vec{k})-\hat{a}^\dagger(-\omega,-\vec{k})\right]\left[\hat{a}(-\omega,-\vec{k}')-\hat{a}^\dagger(\omega,\vec{k}')\right]\\
&\hskip 1.9in{}+c^2\vec{k}\cdot\vec{k}'\left[\hat{a}(\omega,\vec{k})+\hat{a}^\dagger(-\omega,-\vec{k})\right]\left[\hat{a}(-\omega,-\vec{k}')+\hat{a}^\dagger(\omega,\vec{k}')\right]\bigg\}e^{i[(\vec{k}-\vec{k}')\cdot\vec{x}]}\\
&\hskip 2.15in{}+\int d^3xV(\hat{\phi})\\
&=\int\frac{d^3\vec{k}}{(2\pi)^3}\frac{\hbar}{4\omega}\bigg\{\frac{\kappa}{\rho}\left[\hat{a}(\omega,\vec{k})+\hat{a}^\dagger(-\omega,-\vec{k})\right]\left[\hat{a}(-\omega,-\vec{k})+\hat{a}^\dagger(\omega,\vec{k})\right]\\
&\hskip 1.15in{}-\omega^2\left[\hat{a}(\omega,\vec{k})-\hat{a}^\dagger(-\omega,-\vec{k})\right]\left[\hat{a}(-\omega,-\vec{k})-\hat{a}^\dagger(\omega,\vec{k})\right]\\
&\hskip 1.00in{}+c^2\vec{k}^2\left[\hat{a}(\omega,\vec{k})+\hat{a}^\dagger(-\omega,-\vec{k})\right]\left[\hat{a}(-\omega,-\vec{k})+\hat{a}^\dagger(\omega,\vec{k})\right]\bigg\}+\int d^3xV(\hat{\phi})\\
&=\int\frac{d^3\vec{k}}{(2\pi)^3}\frac{\hbar}{4\omega}\bigg\{\left(\frac{\kappa}{\rho}-\omega^2+c^2\vec{k}^2\right)\left[\hat{a}(\omega,\vec{k})\hat{a}(-\omega,-\vec{k})+\hat{a}^\dagger(-\omega,-\vec{k})\hat{a}^\dagger(\omega,\vec{k})\right]\\
&\hskip 1.05in{}+\left(\frac{\kappa}{\rho}+\omega^2+c^2\vec{k}^2\right)\left[\hat{a}(\omega,\vec{k})\hat{a}^\dagger(\omega,\vec{k})+\hat{a}^\dagger(-\omega,-\vec{k})\hat{a}(-\omega,-\vec{k})\right]\bigg\}+\int d^3xV(\hat{\phi})\\
&=\int\frac{d^3\vec{k}}{(2\pi)^3}\frac{\hbar}{4\omega}2\omega^2\left[\hat{a}(\omega,\vec{k})\hat{a}^\dagger(\omega,\vec{k})+\hat{a}^\dagger(-\omega,-\vec{k})\hat{a}(-\omega,-\vec{k})\right]+\int d^3xV(\hat{\phi})\\
&=\int\frac{d^3\vec{k}}{(2\pi)^3}\hbar\omega\left\{\hat{a}^\dagger(\omega,\vec{k})\hat{a}(\omega,\vec{k})+\textstyle\frac{1}{2}[\hat{a}(\omega,\vec{k}),\hat{a}^\dagger(\omega,\vec{k})]\right\}+\int d^3xV(\hat{\phi})\\
&=\int\frac{d^3\vec{k}}{(2\pi)^3}\hbar\omega\left\{\frac{1}{2}+\hat{a}^\dagger(\omega,\vec{k})\hat{a}(\omega,\vec{k})\right\}+\int d^3xV(\hat{\phi}).\end{align*}

Once again, things work out almost magically, and we are able to express the Hamiltonian using creation and annihilation operators. This opens the road to the actual development of a quantum field theory.

So why did I find it necessary to capture here something that can be found in first chapter of every semi-decent quantum field theory textbook? Several reasons.

• First, I wanted to present a consistent treatment of all four cases: the nonrelativistic and relativistic case for both the particle and the field theory.
• Second, I wanted to write down all relevant equations without omitting dimensions. I wanted to write down a Lagrangian density that has the dimensions of energy density, consistent with a scalar field that has the dimensions of length (i.e., a displacement).
• Third, I wanted to spell out some of the details of the derivation that are omitted from nearly all textbooks yet, I am obliged to admit, almost stumped me. That is, once you see the derivation the steps are reasonably trivial, but it is still hard to stumble upon exactly the right way to apply the relevant identities related to Fourier transforms and Dirac deltas.
• Lastly, I find it revealing how this approach can highlight exactly where a quantum theory is introduced. In the particle theory case, it is when we assume that linear combinations of eigenstates also represent physical states of a system, despite the fact that they do not correspond to classical eigenvalues. In the case of a field theory, the transition occurs when we replace Fourier coefficients with operators: implicit in the transition is that once again, linear combinations are included as representing actual physical states of the system.

Note also how none of this has anything to do with interpretations. There is no "collapse of the wave function" or any such nonsense. That stuff happens when we introduce into our consideration a "measurement event", effectively an interaction between the quantum system and a classical instrument, which forces the quantum system into an eigenstate. This eigenstate cannot be predicted from the initial conditions alone, precisely because the classical idealization of the measurement apparatus effectively amounts to an admission of ignorance about its true quantum state.