"When General Hooker's movements were so developed as to make sure of his purpose, repeated calls came to me over the wires to pull away from Suffolk and return to General Lee with all speed. These came from general Lee, and also from Richmond authorities. In reply I despatched that our trains were at the front along the coast collecting supplies; that they would be hurried to our rear, and as soon as safe we would march. The calls became so frequent and urgent, however, that I inquired if we should abandon our trains. To this no answer came; and I was left to the exercise of my own judgement.
"As soon as the trains were safely back, we drew off, marched back to the Blackwater, and thence en route for Richmond and Fredericksburg. Before we reached [Richmond] a telegram came announcing the great battle of victory at Chancellorsville."
Source: From Manassas to Appomattox
Although Longstreet doesn't mention it directly in his memoirs, Secretary of War Seddon advised him that he should not abandon his trains but move with all possible speed to gather them so that he could being his move to rejoin Lee's army. Additionally a letter from Lee to Longstreet written on May 1 -- which I have been unable to find a copy of so as to reproduce it here -- indicates that Lee did not expect that Longstreet could join him before he made contact with Hooker, but that he hoped it could be so. Lack of adequate transportation and the careful act of retreating in the face of the enemy made it impossible to rejoin the army before Lee had already clashed with Hooker.
Commentary
Longstreet's mission in southern Virginia and North Carolina kept him from being present at the battle of Chancellorsville, a point with which some historians have taken issue, claiming that Longstreet was slow and disobeyed orders. Longstreet was in a tenuous position in the face of the enemy. Disengaging immediately from around Suffolk would have left his supply trains unguarded and thus in a position where they could be easily captured. In addition, a quick and unplanned move could have invited the Federals to attack his rear which would have caused an even greater delay.
Longstreet had this to say about his operations: