If the rock is granite, the white inclusions are almost certainly quartz. The OP ruled out ab2's original suggestion of calcite with his scratch test, the results of which he edited into his question.
It would take an expert eye to identify the rock as granite from the photograph the OP provided. However, if the OP found it in northeastern Georgia in Elbert county, it is probably granite. According to the New Georgia Encyclopedia:
Tucked away in the northeastern Georgia Piedmont between the Savannah
and Broad rivers lies the city of Elberton, also known as the granite
capital of the world. Currently, more than 150 sheds and manufacturing
plants turn rough blocks of granite extracted from 45 area quarries
into finished memorials, markers, and mausoleums. These granite
products are subsequently shipped to customers all over the world. At
least one monument or marker made from Elberton's granite has been
placed in all fifty states; the granite is also found in several
foreign countries. In recent years Elberton's stone industry has
produced an average of 250,000 granite markers and monuments annually.
In all, a total of 280 separate companies operated businesses in
granite production or related services in Elberton in 2003.
The reason why one cannot identify it as granite from the picture is that it does not show the characteristic speckled pattern of granite -- speckling from large crystals. See the photos in Geology ABC — How to identify granites.
The OP has done just about everything the link given by @Jan Doggen says he should do. I have two suggestions for the next step. (1) Try to break off a piece of the supposedly granite part with a rock hammer and see if the interior has the characteristic speckling of granite. Downside: this ruins your find. (2) Take your find to one of the many granite companies in Elbert County and ask. Downside: They may be impatient with you.
As for putting it back where you found it, IMO, overkill of Leave No Trace, particularly if you found it in Elbert County. The entire county is sitting on granite, it is extensively quarried and sent around the world. If you will display it and will often admire it, take it; BUT...put it back if you will put it in a closet and forget it.