Since Hiraku was much younger than his older siblings, he thought that he would always look up to them, however everything instantly changed the moment he found out about the curse mark. He started fearing that one day he would end up with it, so the moment he became a Genin, he was out on missions at every single hour of the day if possible. He was barely home, even when training.
He was 13 years old when Neji was born, and around 17 years old when Hizashi died. It was around that time that he found out that the reason he wasn't branded as well was because of Hizashi, who made his parents promise that they wouldn't do something so cruel to Hiraku. As a way to honor his death, and show how grateful he was, he decided to take care of Neji in his stead.
Although he tried as hard as he could to stop the elders and his brother from branding Neji, they did it behind his back when he was on a one month long mission. When Hiraku got back, he almost murdered the entire council of elders of the clan. He was screaming as Hiashi held him back, and then he heard one sentence that turned his hatred into guild.
‘Hizashi had to give something in return when we kept our promise.’
The guilt instantly turned back into hatred when he realised he was being blamed for something he had no control over, and that they were trying to manipulate him.
It was after another long mission, when he came home to Neji crying while holding his head, that he decided that they would not stay in the Hyuga compound anymore. As Hiraku packed his and Neji's things, Hiashi entered their house without knocking. He didn't put up much of a fight, but he did say that Hiraku would never be welcomed to the Hyuga compound anymore if he left, which, to nobody's surprise, Hiraku was more than glad to accept the ‘punishment’.
The Third Hokage, one year later, after many mission reports that made it seem like Hiraku was always one step closer to betraying Konoha and becoming a Missing nin, decided to remove Hiraku from the ANBU forces, and let him have more time in the village with Neji.